Method and Apparatus for Differently Encrypting Different Flows

ABSTRACT

For a host that executes one or more guest virtual machines (GVMs), some embodiments provide a novel encryption method for encrypting the data messages sent by the GVMs. The method initially receives a data message to send for a GVM executing on the host. The method then determines whether it should encrypt the data message based on a set of one or more encryption rules. When the process determines that it should encrypt the received data message, it encrypts the data message and forwards the encrypted data message to its destination; otherwise, the method just forwards the received data message unencrypted to its destination. In some embodiments, the host encrypts differently the data messages for different GVMs that execute on the host. When two different GVMs are part of two different logical overlay networks that are implemented on common network fabric, the method in some embodiments encrypts the data messages exchanged between the GVMs of one logical network differently than the data messages exchanged between the GVMs of another logical network. In some embodiments, the method can also encrypt different types of data messages from the same GVM differently. Also, in some embodiments, the method can dynamically enforce encryption rules in response to dynamically detected events, such as malware infections.

BACKGROUND

Enterprises (e.g., financial service providers, healthcare providers,critical infrastructure providers, etc.) store valuable data, andtransfer it over networks. Information spreads across datacenters oftenthrough dedicated telco-provided networks. Overlay networks provide thesame service across the wide area network (WAN) on a public network forenterprises, and are susceptible to threats such as snooping, man in themiddle attack (MITM), and forging. As enterprises widely adoptcloud-based Software-Defined Data Center (SDDC) instead of dedicateddatacenters, new challenges are introduced, and protecting the dataflowing into, within, and out of the cloud becomes a necessity. Theprivacy guarantee of private datacenters is no longer assumed, andthreats similar to those in the Internet prevail.

Cryptography protects data and communication channels from maliciousparties, provides confidentiality to enterprise dataflow in the cloud,and provides control over the data to the enterprise. However, currentapproaches to encrypt network data fall short in terms of contextualinformation, isolation, granularity, or ease of management. For example,IPSec has been a key game changer in business to business (B2B), virtualprivate networks (VPNs), and branch-office networking. However, IPSectunneling by the edge devices is oblivious to the application contextand the user generating the network traffic, lacks granularity as itencrypts data in bulk, and cannot address various internal threats asthe traffic between the virtual machines (VMs) and the edge devices isin plaintext. Moreover, the known problem with traditional IPSec is thatboth endpoints negotiate a security association, and agree upon a key.In a network that includes N endpoints, the number of keys in the secureoverlay is O(N²).

In-VM encryption provides contextual information for fine-grainedsecurity decisions. However, in-guest encryption fails to isolate theprotected data from the protection mechanism as they both reside in theguest. This scheme also suffers from inability for upgrades, and isextremely difficult to manage from a centralized management console.

BRIEF SUMMARY

For a host that executes one or more guest virtual machines (GVMs), someembodiments provide a novel encryption method for encrypting the datamessages sent by the GVMs. The method initially receives a data messageto send for a GVM executing on the host. The method then determineswhether it should encrypt the data message based on a set of one or moreencryption rules. When the process determines that it should encrypt thereceived data message, it encrypts the data message and forwards theencrypted data message to its destination; otherwise, the method justforwards the received data message unencrypted to its destination.

By employing this method, a host frees up the GVMs from the encryptiontasks, while flexibly encrypting different GVM data messagesdifferently. For instance, in some embodiments, the host encryptsdifferently the data messages for different GVMs that execute on thehost (e.g., encrypts the data messages from one GVM while not encryptingthe data messages for another GVM). When two different GVMs are part oftwo different logical overlay networks that are implemented on a commonnetwork fabric, the method in some embodiments encrypts the datamessages exchanged between the GVMs of one logical network differentlythan the data messages exchanged between the GVMs of another logicalnetwork.

In some embodiments, the method can also encrypt different types of datamessages from the same GVM differently. For example, by examining thedestination identifiers (e.g., destination MAC addresses, destination IPaddresses, etc.) in the header fields of the data messages, the host canencrypt the data messages from a particular GVM to one destination(e.g., a first destination GVM) differently than the data messages fromthe particular GVM to another destination (e.g., a second destinationGVM). By examining the header field values (e.g., the higher-level L4attributes), the host can even differently encrypt different datamessages that are part of different data flows between one pair of GVMs.

The method of some embodiments can dynamically create and enforceencryption rules in response to dynamically detected events. Forinstance, in some embodiments, the GVMs running on the host are scannedfor malware periodically or continuously. When malware is detected on aGVM, the GVM is tagged as an infected GVM. In response to such a tag,the host uses the encryption method of some embodiments to encrypt thedata messages sent by the infected GVM, or to encrypt the data messagessent by the other GVMs on the same host as the infected GVM. In someembodiments, the encryption method also dynamically adds and enforcesencryption rules after detecting different flow-based events.

When the method encrypts a received GVM data message, the encryptedmessage has to be decrypted by a corresponding method at the destinationof the GVM data message. In some embodiments, the encryption anddecryption operations of these two methods are statically synchronizedas they are configured beforehand to use the appropriateencryption/decryption keys for different data messages. For example,when two GVMs on two different hosts operate as part of one logicalnetwork, one or more controllers and/or key managers in some embodimentsprovide the same cryptographic key(s) (e.g., one identical key, or apair of associated keys that are transformed versions of each other) toboth hosts, such that each host uses the provided key(s) to encrypt allGVM data messages at their source and to decrypt all data messages attheir destination. In some of these embodiments, the controllers and/orkey managers periodically refresh the keys that the hosts store fortheir prospective encryption and decryption operations.

In other embodiments, however, the encryption methodology is moredynamic. For instance, in some embodiments, a host's encryption methodinitially detects a condition (e.g., a malware event or a flow-basedevent) that requires data messages from one or more of the host's GVMsto be encrypted. Upon detecting the condition, the host's methodidentifies an encryption key, uses the encryption key to encrypt some orall of the data message (e.g., the payload and some or all of themessage header), and then includes a key identifier (i.e., the key ID)in the data message (e.g., in the message header or in another headerthat is used to encapsulate the message header) so that the datamessage's destination (e.g., the destination host) can know which key touse to decrypt the encrypted message.

The dynamic encryption methodology of some embodiments is ahypervisor-based encryption scheme that uses guest introspection (GI) toget application-centric and contextual metadata about network traffic,in order to dynamically apply attribute-based security policies on theGVM traffic, and encrypt this GVM traffic accordingly. In someembodiments, the hypervisor is a software layer (e.g., an application)over which the GVMs execute. The hypervisor in some embodimentsimplements the above-described encryption and decryption methods.

In some embodiments, the hypervisor also uses guest introspection toobtain application-centric and contextual information about theapplications that each GVM runs. To perform guest introspection, thehypervisor in some embodiments communicates with a thin introspectingagent that is deployed on each GVM. The thin introspecting agent has anetwork introspection module that in some embodiments is called by theTCP/IP stack each time the stack processes a new connection request.Through these calls, the network introspection module captures (1) everynew connection request (e.g., both incoming and outgoing connectionrequests) and (2) contextual information (e.g., user identity andapplication context) for the new connections. In some embodiments, thethin introspecting agent includes other introspection modules thatgather other introspection data. For example, in some embodiments, theagent includes file and system introspection modules that are used toscan the thin agent's GVM for malware.

The above-described hypervisor-based encryption scheme combines the bestof in-guest encryption and network-based encryption. It avoids theoverhead of key management of N-to-N GVMs, and it performs encryptionbased on fine-grained, context aware policies. As such, it providescontext-aware, hypervisor-based network encryption that dynamicallyenforces fine-grained encryption policies, while bridging the gapbetween having sufficient context to perform context-aware encryptionand maintaining isolation between the encryption and the GVM operations.

The preceding Summary is intended to serve as a brief introduction tosome embodiments of the invention. It is not meant to be an introductionor overview of all inventive subject matter disclosed in this document.The Detailed Description that follows and the Drawings that are referredto in the Detailed Description will further describe the embodimentsdescribed in the Summary as well as other embodiments. Accordingly, tounderstand all the embodiments described by this document, a full reviewof the Summary, Detailed Description, the Drawings and the Claims isneeded. Moreover, the claimed subject matters are not to be limited bythe illustrative details in the Summary, Detailed Description and theDrawing.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth in the appendedclaims. However, for purposes of explanation, several embodiments of theinvention are set forth in the following figures.

FIG. 1 illustrates an encryption process for encrypting the datamessages sent by the GVMs.

FIG. 2 illustrates a host encryption architecture of some embodiments.

FIG. 3 illustrates an encryption system of some embodiments.

FIG. 4 illustrates another host encryption architecture of someembodiments.

FIG. 5 illustrates the various introspecting modules of a guestintrospection (GI) agent of some embodiments.

FIG. 6 illustrates examples of various data records stored in anencryption rule data store of some embodiments.

FIG. 7 illustrates various data stores of an encryptor/decryptor of someembodiments.

FIG. 8 illustrates an encryption process of some embodiments of theinvention.

FIGS. 9 and 10 illustrate decryption processes of some embodiments ofthe invention.

FIGS. 11-13 illustrate several examples of the different encryptionscenarios that the encryption architecture of FIG. 4 can achieve.

FIG. 14 illustrates an example of several logical switches that aredefined by multiple software switches that execute on multiple hosts.

FIG. 15 illustrates an encryption scenario in which two different flowsbetween the same pair of GVMs are encrypted differently.

FIGS. 16 and 17 present two examples that illustrate dynamicallyencrypting packets after the detection of malware.

FIGS. 18 and 19 illustrate two processes that are employed in someembodiments to dynamically specify encryption rules for detectedmalware.

FIGS. 20 and 21 illustrate two processes that are employed in someembodiments to dynamically specify encryption rules for dynamicallydetected flow based events.

FIG. 22 presents a more-detailed diagram of an encryption system of someembodiments.

FIGS. 23-25 illustrate the workflow message exchange within theencryption system of some embodiments.

FIG. 26 illustrates a process for dynamically adding an infected GVM toa malware security group, and dynamically applying this' group'sencryption rule to this GVM, until the malware has been resolved, atwhich time the GVM is dynamically removed from the security group.

FIG. 27 illustrates a process for adding a GVM to a security group aftera flow-based event is detected, and removing the GVM from the securitygroup when the flow-based event is terminated.

FIG. 28 illustrates a computer system that is used in some embodimentsto implement a host computer, a controller computer or a managercomputer.

DETAILED DESCRIPTION

In the following detailed description of the invention, numerousdetails, examples, and embodiments of the invention are set forth anddescribed. However, it will be clear and apparent to one skilled in theart that the invention is not limited to the embodiments set forth andthat the invention may be practiced without some of the specific detailsand examples discussed.

For a host computing device (the “host”) that executes one or more guestvirtual machines (GVMs), some embodiments provide a novel encryptionmethod for encrypting the data messages sent by the GVMs. Examples ofGVMs include webservers, application servers, database servers, etc. Insome cases, all the GVMs belong to one entity, e.g., an enterprise thatoperates a datacenter with multiple hosts. In other cases, the hostexecutes in a multi-tenant environment (e.g., in a multi-tenant datacenter), and different groups of GVMs belong to different tenants. Asused in this document, encryption refers to the encoding of “plaintext”data (i.e., unencrypted data) into a “ciphertext” format by using anencryption algorithm (also called encryption operator or process) thattakes as input, the plaintext data and an encryption key. Decryptionrefers to the conversion of the encrypted ciphertext data into plaintextdata by applying a decryption algorithm (also called decryption operatoror process), which relies on a decryption key, on the encryptedciphertext data.

FIG. 1 illustrates a process 100 that implements the novel encryptionmethod of some embodiments of the invention. This process 100 interceptsa data message from a GVM along the message's datapath, encrypts thismessage if necessary, and then sends the encrypted or unencryptedmessage out along its datapath. As used herein, data messages refers toa collection of bits in a particular format sent across a network. Oneof ordinary skill in the art will recognize that the term data messagemay be used herein to refer to various formatted collections of bitsthat may be sent across a network, such as Ethernet frames, IP packets,TCP segments, UDP datagrams, etc.

In some embodiments, the process 100 of FIG. 1 is performed by the hoston which the GVMs execute. This process 100 will be described below byreference to FIG. 2, which illustrates one architecture for a host 200to intercept and encrypt outgoing GVM data messages, and to interceptand decrypt incoming GVM data messages. Specifically, FIG. 2 illustratesa host 200 that executes multiple GVMs 205, a software forwardingelement 210, a set of one or more encryptors/decryptors 215 (referred toas the “encryptor set”), and a malware-detecting SVM 220.

The software forwarding element (SFE) 210 executes on the host tocommunicatively couple the GVMs of the host to each other and to otherdevices (e.g., other GVMs) outside of the host. As shown, the SFE 210includes a port 230 to connect to a physical network interface card(NIC) of the host, and a port 235 to connect to the virtual NIC (VNIC)225 of each GVM. In some embodiments, the VNICs are softwareabstractions of the physical NIC (PNIC) that are implemented by thevirtualization software (e.g., by a hypervisor). Each VNIC isresponsible for exchanging packets between its GVM and the SFE 210through its corresponding SFE port. As shown, a GVM's egress datapathfor its data messages includes (1) the GVM's VNIC 225, (2) the SFE port235 that connects to this VNIC, (3) the SFE 210, and (4) the SFE port230 that connects to the host's PNIC.

Through its port 230 and a NIC driver (not shown), the SFE 210 connectsto the host's PNIC to send outgoing packets and to receive incomingpackets. The SFE 210 performs message-processing operations to forwardmessages that it receives on one of its ports to another one of itsports. For example, in some embodiments, the SFE tries to use headervalues in the GVM data message to match the message to flow based rules,and upon finding a match, to perform the action specified by thematching rule (e.g., to hand the packet to one of its ports 230 or 235,which directs the packet to be supplied to a destination GVM or to thePNIC). In some embodiments, the SFE 210 is a software switch, while inother embodiments it is a software router or a combined softwareswitch/router.

The SFE 210 in some embodiments implements one or more logicalforwarding elements (e.g., logical switches or logical routers) withSFEs executing on other hosts in a multi-host environment. A logicalforwarding element in some embodiments can span multiple hosts toconnect GVMs that execute on different hosts but belong to one logicalnetwork. In other words, different logical forwarding elements can bedefined to specify different logical networks for different users, andeach logical forwarding element can be defined by multiple SFEs onmultiple hosts. Each logical forwarding element isolates the traffic ofthe GVMs of one logical network from the GVMs of another logical networkthat is serviced by another logical forwarding element. A logicalforwarding element can connect GVMs executing on the same host and/ordifferent hosts.

In the host architecture illustrated in FIG. 2, the process 100 isperformed by the SFE ports 235 and the encryptor(s) 215. The ports 235in some embodiments include one or more function calls to one or moremodules that implement special input/output (I/O) operations on incomingand outgoing packets that are received at the ports. One of thesefunction calls for a port is to an encryptor/decryptor in theencryptor/decryptor set 215. As further described below, theencryptor/decryptor performs the encryption/decryption rule check andthe encryption/decryption operations on outgoing/incoming data messages(i.e., on messages that are sent by one of the GVMs and/or that arereceived by the host for one of the GVMs). In some embodiments, eachport 235 has its own encryptor/decryptor 215, while in otherembodiments, some or all of the ports 235 share the sameencryptor/decryptor 215 (e.g., all the ports share oneencryptor/decryptor, or all ports that are part of the same logicalnetwork share one encryptor/decryptor).

Examples of other I/O operations that are implemented by the ports 235include message encapsulation operations needed for sending messagesalong tunnels to implement overlay logical network operations. Byimplementing a stack of such function calls, the ports can implement achain of I/O operations on incoming and/or outgoing messages in someembodiments. Instead of calling the I/O operators (including theencryptor set 215) from the ports 235, other embodiments call theseoperators from the GVM's VNIC or from the port 230 of the SFE. Forinstance, as further described below, the port 230 in some embodimentscalls a decryptor when it receives an encrypted GVM message from outsideof its host and the GVM message has its L2 payload and L2 header valuesencrypted. The port 230 in some embodiments calls the decryptor 215 todecrypt this L2 encrypted message so that it can obtain the L2 headervalue that the SFE 210 needs to identify the port 235 to which it needsto pass the GVM data message.

As shown in FIG. 1, the process 100 initially receives (at 105) a datamessage to send from a GVM executing on the host. The process 100 thendetermines (at 110) whether it should encrypt the data message based ona set of one or more encryption rules. In FIG. 2, this determination insome embodiments is made (at 110) by (1) a port 235 relaying to itsencryptor 215 the GVM data message that the port 235 receives from theVNIC 225 of the GVM that sent the message, and (2) the encryptor 215using the GVM message's attributes to examine encryption rules that arestored in an encryption rule data store 250 of the host. In someembodiments, the port 235 relays the GVM message by passing to theencryptor 215 a reference (e.g., a handle that identifies a location inmemory that stores the GVM message) to the GVM message.

The GVM message attributes that the encryptor/decryptor uses in someembodiments to check the encryption/decryption rules include the messageheader values. For instance, the encryptor in some embodimentsdetermines whether it should encrypt a GVM data message by using thedata message's header values (e.g., its L2-L4 attributes) to identify anencryption rule that is applicable to the GVM data message. For an L2frame, the header values include the source and destination MACaddresses, while for an L3/L4 packet, the header values are the fivetuple identifiers, which include the packet's source identifier,destination identifier, source port, destination port, and protocol(service). Also, in some embodiments, one or more of the identifiervalues can be logical values that are defined for a logical network(e.g., can be virtual network identifier (VNI) for a VXLAN overlaynetwork, or a logical IP addresses defined in a logical address space).In other embodiments, all of the identifier values are defined in thephysical domains. In still other embodiments, some of the identifiervalues are defined in logical domain, while other identifier values aredefined in the physical domain.

To determine (at 110) whether the GVM data message should be encrypted,the encryptor 215 uses the GVM message attributes to examine theencryption rules stored in the encryption rule data store 250, in orderto determine whether this data store 250 contains a rule that identifiesan encryption key for encrypting the received GVM data message.Similarly, to decrypt at least some of the encrypted data messages, adecryptor 215 in some embodiments uses the message attributes of theseencrypted data messages to search the encryption rules stored in theencryption rule data store 250 to identify a rule that identifies a keyfor decrypting the received GVM data message. In some embodiments, thedecryptor 215 identifies a key for decrypting a received encrypted GVMdata message by using a key identifier that is inserted in the GVM datamessage by the encryptor of this message or by another module at thedirection of this encryptor. The processes for decrypting encryptedmessages will be further described below.

When the encryptor 215 determines (at 110) that it should encrypt theGVM data message, it (at 115) retrieves the GVM data message from memory(e.g., at a location provided by the SFE port or GVM VNIC that calledthe encryptor), encrypts the data message and passes the encrypted datamessage back to the SFE port 235 that called it. The SFE port 235 thensends the encrypted data message along the message's datapath.

When the encryptor 215 determines (at 110) that it should not encryptthe GVM data message, it directs (at 120) the module that called it topass the GVM data message unencrypted along its datapath. In the exampleillustrated in FIG. 2, this operation entails informing the SFE port 235to pass the GVM data message to the SFE 210 so that the SFE can processthe GVM data message to forward the message to its intended destination.After 115 or 120, the process 100 ends.

By employing the process 100, the host 200 frees up the GVMs from theencryption tasks, while flexibly encrypting the GVM data messages basedon any number of encryption rules in the encryption rule data store 250.For instance, based on these rules, the host 200 can encrypt differentlythe data messages for different GVMs that execute on it. One example ofdifferent encryption involves encrypting the data messages from one GVMwhile not encrypting the data messages for another GVM. Other examplesof different encryption include encrypting different portions of the GVMmessages (e.g., encrypting just the L2 payload, encrypting the L2 headerand payload, etc.), using different encryption algorithms (e.g., usingAES128GCM encryption versus using AES256GCM encryption, or using AES-GCMencryption versus using AES-CBC/SHA1-HMAC encryption, or using AESencryption versus using 3DES encryption), using different encryptionkeys, etc. Also, as further described below, some embodiments not onlyencrypt the payload of the GVM message, but also (1) perform integritycheck value (ICV) calculations on the GVM message payload and some orall of the unencrypted header values (e.g., L3/L4 header values, orlogical network identifier values), and then (2) encrypt the hash valuethat results from the ICV calculation along with the payload. Afterdecrypting the payload along with the ICV-generated hash, thedestination host can then authenticate the GVM data message byauthenticating the hash value for the GVM data message's header valueand payload. The encryption of the payload and the ICV-generated hashwill be further described below.

When two different GVMs are part of two different logical overlaynetworks that are implemented on common network fabric (e.g., on acommon set of host SFEs and one or more intermediate forwardingelements), the process 100 in some embodiments encrypts the datamessages exchanged between the GVMs of one logical network differentlythan the data messages exchanged between the GVMs of another logicalnetwork.

In some embodiments, the process can also encrypt different types ofdata messages from the same GVM differently. For example, by examiningthe destination identifiers (e.g., destination MAC addresses,destination IP addresses, etc.) in the header fields of the datamessages, the host can encrypt the data messages from a particular GVMto one destination (e.g., a first destination GVM) differently than thedata messages from the particular GVM to another destination (e.g., asecond destination GVM). By examining the header field values (e.g., thehigher-level L4 attributes), the host can even encrypt differentlydifferent data messages from one GVM to another GVM that are part ofdifferent data flows.

In some embodiments, the malware-detecting SVM 220 of the host scans theGVMs of the host for malware periodically or continuously. Malware ismalicious software used to disrupt computer operations, gather sensitiveinformation, and/or gain access to computer systems. Malware can appearin the form of executable code, scripts, active content, and othersoftware. Examples of malware include viruses, spyware, worms, adware,Trojan horses, and other malicious programs

When the SVM detects malware on a GVM, the SVM 220 tags the GVM as aninfected GVM. In response to such a tag, the host uses the encryptionprocess of some embodiments to encrypt the data messages sent by theinfected GVM, or to encrypt the data messages sent by the other GVMs onthe same host as the infected GVM, or both. As further described below,the host 200 performs this encryption by (1) defining an encryption rulefor this event, (2) using encryption policies that it stores to convertthe security tag to a set of L2-L4 parameters that specify matchingattributes for the encryption rule, and (3) pushing this rule into theencryption rule data store 250. This dynamically created rule in someembodiments requires the encryptor or another module to include a keyidentifier with the encrypted GVM data message, so that the destinationhost can know what key to use to decrypt the encrypted GVM data message.

The creation of an encryption rule in response malware-related event isone example of how the encryption architecture of some embodiments candynamically create and enforce encryption rules in response to adetected event. In some embodiments, the encryption architecture canalso dynamically add encryption rules to the encryption rule data storeafter detecting different flow-based events. The dynamic addition ofsuch encryption rules will be further described by reference to FIG. 4,which presents a more detailed encryption architecture of someembodiments.

However, before describing this more-detailed architecture, theencryption system 300 of some embodiments will be described by referenceto FIG. 3. As shown, this system includes multiple virtualized hosts305-315, a set of controllers 320, and a set of key managers 325. Thevirtualized hosts 305-315 are similar to the host 200 of FIG. 2, exceptthat the hosts 305-315 each are shown to include an encryption agent 360for interacting with the controller set 320 and the key manager set 325.In FIG. 3, the SVM 220, ports 230 and 235, VNICs 225, and the rule datastore 250 are not shown in order to keep this figure's illustrationsimple. As shown in FIG. 3, the hosts 305-315, the controller set 320,and the key manager set 325 communicatively couple through a network375, which can include a local area network (LAN), a wide area network(WAN) or a network of networks (e.g., Internet).

The network controllers 320 provide control and management functionalityfor defining and managing the instantiation of one or more GVMs on eachhost. These controllers in some embodiments also provide control andmanagement functionality for defining and managing multiple logicalnetworks that are defined on the common software forwarding elements ofthe hosts. In some embodiments, these controller 320 also createsecurity groups, security policies (including encryption policies) andencryption rules. The key managers provide encryption keys for thevarious encryption rules that the hosts enforce. In some embodiments,the key managers also periodically provide new encryption keys for oneor more of the encryption rules, in order to make it harder for thirdparties to break the encryption scheme.

In some embodiments, the hosts for the source and destination GVMs of aGVM data message use encryption and decryption operations that arestatically synchronized as they are configured beforehand to use theappropriate encryption/decryption keys for different data messages. Forexample, when GVMs 350 and 352 on hosts 305 and 310 are part of onelogical network, the controller set 320 or key manager set 325 providesthe same cryptographic key(s) (e.g., one identical key, or a pair ofassociated keys that are transformed versions of each other) to bothhosts 305 and 310, such that each host uses the provided key(s) toencrypt all data messages that it receives from it own GVM 350 or 352,and to decrypt all data messages that it receives from the other host'sGVM 352 or 350. In some of these embodiments, the controllers and/or keymanagers periodically refresh the keys that the hosts store for theirprospective encryption and decryption operations.

In other embodiments, however, the encryption process is more dynamic.For instance, in some embodiments, the encryption process 100 of thehost (e.g., host 305) initially detects a condition that requires datamessages form one or more of the host's GVMs (e.g., GVM 350) to beencrypted. This detection can be based on an analysis of the headervalues of the GVM messages and/or based on a malware scan of the GVM(s).Upon detecting the condition, the host's encryption process identifiesan encryption key, uses the encryption key to encrypt some or all of thedata message (e.g., the payload and some or all of the message header),and then includes a key identifier (i.e., the key ID) in the datamessage (e.g., in the message header or in another header that is usedto encapsulate the message header) so that the data message'sdestination (e.g., the destination host 310) can know which key to useto decrypt the encrypted message.

The dynamic encryption process of some embodiments is a hypervisor-basedencryption scheme that uses guest introspection (GI) to getapplication-centric and contextual metadata about network traffic, inorder to dynamically apply attribute-based security policies on the GVMtraffic, and encrypt the GVM traffic accordingly. FIG. 4 illustrates anexample of a hypervisor-based encryption architecture 402 of someembodiments. Specifically, this figure illustrates a host 400 that hasmany of the same modules as the host 200 of FIGS. 2 and 3, such as theGVMs 205, VNICs 225, SFE 210, SFE ports 230 and 235, encryptor set 215,the SVM 220, encryption rule data store 250, and encryption agent 360.

As shown in FIG. 4, the host 400 also includes the hypervisor 405, anencryption state cache 410, key data stores 415 and 420, an encryptionrule data store 425, a flow-based event detecting SVM 465 and guestintrospectors 430. The hypervisor 405 is a software layer (e.g., anapplication) over which the GVMs 205 and other host encryption modulesexecute. As further described below, these encryption modules allow thehost to encrypt and decrypt GVM data messages based on static and/ordynamic encryption schemes. The encryption modules include thepreviously-described encryptor/decryptor set 215, the encryption ruledata store 250, the SVMs 220, and the encryption agent 360. Thesemodules also include encryption state cache 410, the key stores 415 and420, encryption rule data store 425, and the flow-based event detectingSVM 465, which will be further described below.

To facilitate its dynamic encryption operations, the hypervisor 405 usesguest introspection to obtain application-centric and contextualinformation about the applications that each GVM runs. In someembodiments, guest introspection provides information about sensitivedata, the applications and the users accessing them, and where and howit flows in the network (e.g., the source port associated with theconnection session of the application that initiated a flow). It alsoallows the encryption system to identify the endpoints between which thenetwork traffic needs to be secured, provides a way to isolate thetraffic cryptographically, and protects it from the sharedinfrastructure.

To perform guest introspection, the hypervisor in some embodimentscommunicates with the guest introspectors 430. In some embodiments, eachguest introspector 430 is a thin introspecting agent that is deployed ona GVM 205. FIG. 5 illustrates the various introspecting modules of aguest introspection (GI) agent 430 of some embodiments. As shown, theintrospecting modules of the GI agent 430 includes a file introspector505, a system introspector 510, and a network introspector 515. In someembodiments, the flow-based event detecting SVM 465 uses the networkintrospector 515 to obtain real-time data for the real-time analysisthat allows the SVM 465 to report to the encryption agent 360 newflow-based events in real-time, so that this agent can push encryptionrules in real-time to the encryption rule data store 250 for newlydetected flow-based events. Alternatively, in some embodiments, themalware-detecting SVM 220 uses the file and system introspectors 505 and510 to gather batch data (i.e., non-real-time data) in an offline mode,which allows this SVM to report malware events to the encryption agent360. For these malware events, the encryption agent 360 can then pushencryption rules to the data store 250 that direct the encryptor set 215to encrypt the messages of the infected GVM and/or the messages of theother GVMs on the same host.

As shown in FIG. 5, the thin agent uses a multiplexing module 560 thatreceives introspection messages from the various introspectors 505, 510,and 515, and sends these messages to the correct destination (e.g., tothe correct SVM 220 or 465). In some embodiments, the introspectorsprovide the introspection messages to the multiplexor 560 through a VMcommunication interface (e.g., the VMCI interface of VMware Inc.).

The network introspector 515 of the introspecting agent 430 in someembodiments is called by the GVM's TCP/IP stack each time the stackprocesses a initiates or terminates a connection request. Through thesecalls, the network introspection module captures (1) every newconnection request (e.g., both incoming and outgoing connectionrequests) that is made by an application 555 that is operating on theGVM 205, and (2) contextual information (e.g., user identity,application context, etc.) for the new connections. In some embodiments,different flows are differentiated based on the source port that isassociated with the connection session that is associated with the flow.

For outgoing connections, the network inspector in some embodiments canprecisely identify which user initiated the connection, including theActive Directory (AD) groups of which the user is a member. Forinstance, if different users from the Finance and Human Resources groupsof an enterprise are logged in on a terminal server, the networkintrospector can identify which user from which group initiated aparticular network connection. Also, in some embodiments, the networkintrospector provides the application context with each networkconnection initiated or accepted by a GVM. For instance, the networkintrospector in some embodiments provides information about theapplication associated with every outgoing connection. For incomingconnections, it provides detailed information on the listeningapplication in some embodiments. This information in some embodimentsincludes name of the process, application hash, publisher, etc. Thenetwork introspector enables the gathering of this information withoutthe need to do costly deep packet introspection on the received GVM datamessages.

Through the multiplexor 560, the thin guest-introspecting agent 430 insome embodiments provides the captured connection sessions and theirassociated contextual metadata to the flow-based event detecting SVM465. In some embodiments, the SVM 465 then examines its configurationand cache stores (not shown) to determine whether the captured event isan event for which it should provide a notification to the encryptionagent 360, and if so, whether it has previously provided such anotification to the encryption agent 360. The SVM 465 notifies theencryption agent 360 of a flow-based event when the SVM determines thatthe event is one for which the encryption agent 360 needs to receive anotification. As further described below, the agent then examines itsencryption policies for the detected flow-based event, and if necessary,creates and pushes encryption rule(s) to the encryption data store 250to address the newly detected flow-based event. Based on the pushedrule(s), the encryptor set 215 then encrypts the data messages relatedto the detected flow-based event.

The file and system introspectors 505 and 510 are used by themalware-detecting SVM 220 to scan the thin agent's GVM for malware. Insome embodiments, the file and system introspectors are implemented likethe endpoint introspecting agents of the vShield product of VMware Inc.Information about these introspector can be found at:

http://www.vmware.com/files/pdf/vmware-vshield-endpoint-ds-en.pdf

http://www.vmware.com/pdf/vshield_(—)51_admin.pdf

When this SVM 220 detects a malware event on a GVM, the SVM in someembodiments assigns the GVM with the appropriate malware tag (e.g., aninfected tag) in its data store 440. The encryption agent 360 thennotices this malware tag (e.g., by receiving notification from the SVM220 or by receiving a call back from the security-tag data store 440).As further described below, the agent then examines its encryptionpolicies for the malware tag, and if necessary, creates and pushesencryption rule(s) to the encryption data store 250 for the detectedmalware condition. Based on the pushed rule(s), the encryptor set 215then encrypts the data messages sent by the infected GVM, or the datamessages sent by the other GVMs on the host, or both.

As shown in FIG. 4, the encryption agent 360 includes acontroller/manager interface 452, an encryption-agent processor 450, anda publisher 454. The controller/manager interface 452 is responsible forhandling the communication with the controller set 320 and the keymanager set 325. The agent processor 450 processes events that itreceives from the malware-detecting SVM 220 and the flow-basedevent-detecting SVM 465. Specifically, after receiving one such event,the processor 450 examines encryption policies that it stores in 456 todetermine whether it should specify an encryption rule for the receivedevent. The processor 450 receives the encryption policies that it storesin the policy store 456, from the controller set 320.

When the processor 450 determines that it should specify an encryptionrule for a received event, the processor 450 checks the agent'sencryption rule data store 425 to determine whether it already containsthis rule. If not, the processor 450 specifies the encryption rule andstores this rule in the data store 425. In some embodiments, theencryption processor 450 also receives encryption rules from thecontroller set 320, and stores these rules in the encryption data store425. The processor 450, in some embodiments, supplies event data (e.g.,after receiving a flow-based event or a malware event from the SVM 220or 465) to the controller set 320 through the interface 452. For some ofthe reported events, the processor 450 receives encryption rules fromthe controller set 320 for the supplied event data.

For instance, when the processor reports a flow-based event that adoctor has logged onto an application on a first source GVM and thatthis application is in communication session with another application ona second destination GVM, the controller set 320 may detect that a nursehas logged onto the other second-GVM application and push an encryptionrule to the processor so that the flows from the first GVM's applicationto the second GVM's application can be encrypted. In some embodiments,the processor 450 also receives encryption rules from the controller set320. Also, when a GVM is powered on and this GVM is part of a securitygroup with an associated encryption security policy, the controller set320 will create encryption rules and provide them to the processor 450.In some embodiments, the processor 450 also receives from the controllerset, encryption policies after the processor reports the occurrence of amalware or flow-based event. The processor 450 stores the receivedencryption policies in its policy data store 456.

When the processor 450 receives or specifies an encryption rule for itsdata store 425, it contacts the key manager set 325 (through theinterface 452) and obtains the key that is identified by the keyidentifier of the encryption rule. The processor 450 then stores thereceived key in its key data store 420. In some embodiments, theencryption processor 450 also interacts with the controller set 320 andthe key manager set 325 to obtain encryption policies, rules and keys inan offline-batch mode. For instance, the processor 450 may receive suchpolicies, rules and keys when the encryption agent 360 is initiallybeing configured. Also, it can receive such policies, rules, and keyswhen a GVM, a logical forwarding element or logical network is beinginstantiated on the agent's host. Also, the processor 450 receives keyupdates periodically or on-demand in order to perform key rotation onone or more encryption keys (e.g., add or remove keys) in the data store420.

The publisher 454 publishes new encryption rules and their associatedkeys from the data stores 425 and 420 to the encryption rule data store250 and key data store 415 of the encryptor set 215. Once the encryptionrule has been pushed into the data store 250 by the encryption agent360, the rule can then be retrieved by the encryptor set 215 so that therule's associated key can be used to encrypt GVM data messages that aresubsequently received from the GVM with the newly created connectionsession or the GVM with the malware tag. In some embodiments, theencryptor set 215 retrieves the rule's associated key from the keystorage 415 based on the key identifier that is provided by the rule.

To find the encryption rule, the encryptor set 215 first examines itsencryption state cache 410 that stores all the encryption rules that theencryptor set 215 has recently used to encrypt data messages. As furtherdescribed below, this cache 410 is smaller and is addressed differentlythan the encryption data store 250, and therefore is faster to searchthan the encryption data store 250.

If the encryptor set 215 does not find an encryption rule in itsencryption cache 410, it then examines the encryption rules in itsencryption rule data store 250. As shown in FIG. 6, the encryption ruledata store 250, in some embodiments, can include logical-networkencryption rules 605, security-tag-related encryption rules 610, andother general flow-based encryption rules 615. In some embodiments, thelogical-network encryption rules 605 specify different encryption keysfor differently encrypting the GVM data messages of different logicalnetworks. Security-tag-related encryption rules 610 in some embodimentsspecify encryption rules for encrypting data messages from GVMs thathave been tagged as infected and/or GVMs that operate on hosts with atleast one GVM that has been tagged as infected. In some embodiments,flow-based encryption rules 615 specify encryption rules for staticallyor dynamically detected flow-based events.

As shown in FIG. 6, each rule 605, 610, or 615 has a rule identifier set620 and a rule attribute set 625. For the logical-network rules 605, therule identifiers 620 in some embodiments are logical network identifiers(e.g., virtual network identifiers (VNIs) of VXLAN-based logicalnetworks, or virtual distributed router identifiers (VDRIs) of a logicalrouter). For the security-based rules 610 and the general flow-basedrules 615, the rule identifiers 620 in some embodiments are arbitraryL2, L3, and L4 header parameters. In some embodiments, the source port(L4 parameter) is used to differentiate GVM data message from differentflows between the same two GVMs.

As shown, the rule attribute sets 625 for each of these rule typesspecifies (1) an encryption type 630 that specifies the type ofencryption/decryption to use, and (2) a key identifier 635 thatidentifies the key to use for the encryption/decryption. In someembodiments, the rule attribute sets 625 only specify the keyidentifiers 635 and do not specify the encryption types 630, because thekey identifiers identify both the key and the type ofencryption/decryption. In some embodiments, the rule data store 425stores its encryption rules in the same format as that shown in FIG. 6for the rule data store 250.

In some embodiments, rule data store 250 or 425 is searched (e.g., bythe encryptor set 215 or the agent 360) by comparing one or more GVMmessage attributes (e.g., message header values) to the rule identifiersets to identify a rule that has a rule identifier set that matches theGVM message attributes. In some of these embodiments, the rules arestored in the rule data store in a hierarchical way so that when tworules potentially match a GVM message attribute set, the higher priorityrule appears first in the order that the rule data store records aresearched. Also, in some embodiments, the encryption rule data store hasa default rule that is used when no other rule matches a GVM messageattribute set; this rule specifies no encryption key as no rule existsfor encrypting the GVM data message for the received attribute set. Insome embodiments, when the default rule is returned to the encryptor set215, the encryptor set 215 does not encrypt the GVM data message forwhich it is performing the check.

In some embodiments, the encryption-state cache 410 stores theencryption rules based on hashed address values. These hashed addressvalues are hashed versions of the messaged-attribute sets 620 that areused to store the encryption rules in the encryption-rule data store250. The hash address values specify memory locations in the cache 410that store the corresponding message-attribute sets. Because of thisaddressing scheme, the encryptor can search for matching records muchfaster in the cache than in the rule data store 250.

FIG. 7 illustrates that in some embodiments each encryptor/decryptorpair has its own rules data store 250, encryption state cache 410 andkey data store 415. As mentioned above, each SFE port has its ownencryptor/decryptor in some embodiments. In FIG. 7, eachencryptor/decryptor pair is for one SFE port.

The above-described hypervisor-based encryption scheme combines the bestof in-guest encryption and network-based encryption. It avoids theoverhead of key management of N-to-N guests, while providing encryptionbased on fine-grained, context aware policies. As a context-aware,hypervisor-based network encryption that dynamically enforcesfine-grained encryption policies, it bridges the gap between havingsufficient context to make intelligent security decisions (e.g., toperform context-aware encryption) and maintaining isolation between theencryption and the GVM operations. It uses guest introspection toidentify sensitive data flow, gather application-centric and contextualinformation, which it then uses to perform on demand attribute basedencryption.

The encryption and decryption operations of the encryptor set 215 willnow be described by reference to FIGS. 8-10. FIG. 8 illustrates aprocess 800 that the encryptor set 215 performs to encrypt a GVM datamessage. In some embodiments, an encryptor in the encryptor set 215performs this operation when its corresponding SFE port 235 calls theencryptor to check whether a data message from the port's GVM should beencrypted, and if so, to encrypt this message.

As shown, the process 800 initially identifies (at 805) a set of messageattributes that the process uses to identify an encryption rule that isapplicable to the received GVM data message. For different types ofencryption rules, the message-attribute sets that are used to retrievethe rules can be different. For instance, when different encryptionrules are specified for different logical network constructs (e.g., fordifferent logical forwarding elements (such as logical switches, logicalrouters, etc.), logical networks, etc.), the encryption rules are storedin the data stores (e.g., data stores 250 or 425) based on logicalnetwork-construct identifiers, and the message-attributes sets are thelogical identifiers (e.g., the VNIs, VDRIs, the logical MAC addresses,the logical IP addresses, etc.) that are specified in the GVM datamessages. When the encryption rules are specified for particular orarbitrary combination of L2-L4 header values, the message-attribute setscan include any combination of the L2-L4 header values of the GVM datamessage.

After 805, the encryptor determines (at 810) whether its encryptionstate cache 410 stores a cached encryption rule for the identified setof message attributes. As mentioned above, each time an encryptor findsan encryption rule for a GVM data message in some embodiments, it storesa copy of the encryption rule in the encryption state cache 410, so thatwhen it receives another GVM data message with the same identifiedmessage-attribute set (e.g., when it received another GVM data messagethat is part of the same data flow as the original GVM data message),the encryptor does not have to search the encryption rule data store 250to identify an encryption rule for the subsequently received GVM datamessage. In some embodiments, the encryption-state cache 410 stores theencryption rules based on hashed address values that are hashed versionsof the messaged-attribute sets that are used to store the encryptionrules in the encryption-rule data store 250. This addressing schemeallows the encryptor to search the cache faster than the rule data store250. Hence, before searching the rule data store 250, the encryptorfirst generates a hash value from the message-attribute set identifiedat 805, and then uses this hash value to determine whether the cachestores a matching encryption rule for the received GVM data message.

When the process 800 identifies (at 810) an encryption rule for thereceived GVM data message is in the cache 410, the process (at 815) thenretrieves a key identifier from the identified rule, uses thisidentifier to retrieve a key from the key data store 415, and encryptsthe received GVM data message with the retrieved key. In someembodiments, the process encrypts (at 815) the GVM data message'spayload (e.g., the L2 payload) by using the identified encryption key,while generating an ICV hash of the payload and some or all of theheader values (e.g., the physical L3 and L4 header values and/or logicalL2 or L3 header values), so that the message's destination would have(1) to decrypt the encrypted portion of the GVM message, and (2) toverify the authenticity and integrity of the payload and header valuesthat were used for the ICV calculation.

For some or all of the GVM data messages, the encryption process 800 insome embodiments also encrypts (at 815) a portion of the GVM datamessage header. For GVM messages that are exchanged between machinesassociated with a logical network, some embodiments encrypt all of thephysical header values of the GVM message, as they use the logicalnetwork identifier (e.g., the VNI) of GVM to identify the key fordecrypting the encrypted GVM message. Some of these embodiments performICV operation on the logical network identifier (e.g., the VNI) and thepayload so that the decryptor at the destination host can verify theauthenticity and integrity of the encrypted GVM message.

After encrypting the GVM data message, the process (at 815) sends theencrypted GVM data message along its datapath. In some embodiments, thisoperation entails returning a communication to the SFE port 235 (thatcalled the encryptor to initiate the process 800) to let the port knowthat the encryptor is done with its processing of the GVM data message.The SFE port 235 can then handoff the GVM data message to the SFE 210 orcan call another I/O chain operator to perform another operation on theGVM data message.

When the cached encryption rule that was identified at 810 is a rulethat was dynamically created (e.g., by the event detecting SVM 465)after dynamically detecting an event (e.g., after dynamic detection ofthe start of a connection session or dynamic detection of a malwareevent), the encryptor has to make sure (at 815) that the key identifierfor the key that is used to encrypt the GVM message is included in theGVM message header before it is sent. The process 800 accomplishes thisgoal differently in different embodiments. In some embodiments, theprocess 800 passes (at 815) the key identifier to the SFE port 235 (thatcalled it) so that the port or an I/O chain operator that it calls caninsert the key identifier in the GVM message header. For instance, insome embodiments, one I/O chain operator encapsulates the GVM messagewith a header for a logical network identifier that is used to establishan overlay logical network. In some of these embodiments, the SFE port235 passes the key identifier that it receives from the process 800 tothe encapsulating I/O chain operator so that it can include this keyidentifier in its header. In other embodiments, the process 800 insertsthe key identifier in the GVM message header. In some embodiments, theprocess 800 encapsulates the GVM message with a header for a logicalnetwork, and in some of these embodiments, the process inserts the keyidentifier in the encapsulating header values. The marking of the keyidentifier in the GVM message header will be further described below.

After 815, the process 800 ends.

When the process 800 determines (at 810) that the encryption cache 410does not store an encryption rule for the received GVM data message, theprocess 800 searches (at 820) the encryption rule data store 250 toidentify an encryption rule that matches the received GVM message'sattribute set identified at 805. In some embodiments, the encryptionrule data store has a default encryption rule that matches all GVM datamessage, and is returned when no other encryption rule matches theattribute set of a received GVM message. The default encryption rulespecifies that the received data message should not be encrypted (e.g.,specifies a default key identifier that corresponds to a no-encryptionoperation). On the other hand, non-default encryption rules in theencryption data store 250 specify key identifiers that identify keys forencrypting and decrypting the GVM data message.

After 820, the process determines (at 825) whether it was able toidentify an encryption rule that specifies an encryption key forencrypting the received GVM data message. If not (e.g., if the processonly identified a default encryption rule that does not provide anencryption key), the process sends (at 830) the message unencryptedalong the message's datapath. This operation 830 entails informing itsSFE port 235 that it has completed processing the GVM data message.After 830, the process transitions to 840, where in the encryption cachedata store 410, it creates a record to indicate that no encryptionshould be performed for the received GVM data message. In someembodiments, this record is addressed in the cache 410 based on a hashvalue of the message-attribute set identified at 805. The process 800does not create a record in the cache data store 410 when it determinesthat a GVM data message should not be encrypted.

When the process determines (at 825) that it was able to identify (at820) an encryption rule that specifies an encryption key, the processthen (at 835) retrieves a key identifier from the identified rule, usesthis identifier to retrieve a key from the key data store 415, andencrypts the received GVM data message with the retrieved key. Thisencryption of the GVM data message (at 835) is identical to theencryption operation 815 that was above described. For instance, asdescribed above, the process 800 encrypts the GVM data message's payload(e.g., the L2 payload) by using the identified encryption key, whileperforming ICV operation on the payload and some or all of the headervalues (e.g., the physical L3 and L4 header values, logical L2 or L3header values, and/or the logical network identifiers, such as VNIs andVDRIs). For some or all of the GVM data messages, the encryption process800 in some embodiments also encrypts (at 835) some or all of the GVMdata message header (e.g., encrypts some or all of the physical headervalues when the logical network identifier, such as the VNI, is used toidentify the key for decrypting the encrypted GVM message).

After encrypting the GVM data message, the process (at 835) sends theencrypted GVM data message along its datapath. Again, in someembodiments, this operation entails returning a communication to the SFEport 235 (that called the encryptor to initiate the process 800) to letthe port know that the encryptor is done with its processing of the GVMdata message. The SFE port 235 can then handoff the GVM data message tothe SFE 210 or can call another I/O chain operator to perform anotheroperation on the GVM data message.

When the cached encryption rule that was identified at 810 is a rulethat was dynamically created (e.g., by the event detecting SVM 465)after dynamically detecting an event (e.g., after dynamic detection ofthe start of a connection session or dynamic detection of a malwareevent), the encryptor has to make sure (at 835) that the key identifierfor the key that is used to encrypt the GVM message is included in theGVM message header before it is sent. The different manners foraccomplishing this goal in different embodiments were described abovewhen the operation 815 was being described.

After 835, the process transitions to 840, where in the encryption cachedata store 410, it creates a record to indicate that encryption key thatshould be used to encrypt GVM data message with message-attribute setsthat are similar to the set identified at 805. In some embodiments, thisrecord is addressed in the cache 410 based on a hash value of themessage-attribute set identified at 805. After 840, the process ends.

FIG. 9 illustrates a process 900 that the encryptor set 215 performs todecrypt an encrypted GVM data message that a SFE port 235 receives. Insome embodiments, an decryptor in the encryptor set 215 performs thisoperation when its corresponding SFE port 235 calls the decryptor tocheck whether a received GVM data message is encrypted, and if so, todecrypt this message. In some embodiments, the decryptor performs thisprocess only if the header value of the received GVM message does notspecify a key identifier that identifies a key for decrypting the GVMmessage. When the header value does specify such a key identifier, thedecryptor uses a decryption process 1000 of FIG. 10, which will bedescribed below. Also, in some embodiments, the received GVM datamessage has a value (e.g., a bit) that specifies whether the message isencrypted. By analyzing this value, the decryptor will know whether themessage is encrypted. When this value specifies that the message is notencrypted, the decryptor does not call either the process 900 or 1000 todecrypt the encrypted message. Instead, the decryptor informs the SFEport that it can send the GVM data message along its datapath.

As shown, the process 900 initially identifies (at 905) a set of messageattributes that the process uses to identify an encryption rule that isapplicable to the received GVM data message. For different types ofencryption rules, the message-attribute set that is used to retrieve therule can be different. Several examples of such message-attribute setsfor different types of encryption rules were provided above whiledescribing the process 800 of FIG. 8. These examples are equallyapplicable to the discussion of the process 900 of FIG. 9.

After 905, the decryptor determines (at 910) whether its encryptionstate cache 410 stores a cached encryption rule for the identified setof message attributes. Like an encryptor, each time a decryptor finds anencryption rule for a GVM data message in some embodiments, it stores acopy of the encryption rule in the encryption state cache 410, so thatwhen it receives another GVM data message with the same identifiedmessage-attribute set (e.g., when it received another GVM data messagethat is part of the same data flow as the original GVM data message),the decryptor does not have to search the encryption rule data store(s)to identify an encryption rule for the subsequently received GVM datamessage. As mentioned above, the encryption-state cache 410, in someembodiments, stores the encryption rules based on hashed address valuesthat are hashed versions of the messaged-attribute sets that are used tostore the encryption rules in the encryption-rule data store.Accordingly, before searching the rule data store 250, the decryptor insome embodiments first generates a hash value from the message-attributeset identified at 905, and then uses this hash value to determinewhether the cache stores a matching encryption rule for the received GVMdata message.

When the process 900 identifies (at 910) an encryption rule for thereceived GVM data message in the cache 410, the process (at 915) thenretrieves a key identifier from the identified rule, uses thisidentifier to retrieve a key from the key data store 415, and decryptsthe encrypted portion of the received GVM data message with theretrieved key. In some embodiments, part of the decryption operation (at915) is to authenticate the ICV generated hash of the GVM message headerand payload. Specifically, when a portion of the received GVM datamessage (e.g., its physical (e.g., L3 or L4) header values, or itslogical (e.g., VNI) header values) is hashed along with the payloadthrough an ICV operation by the encryptor, the decryption operationverifies this portion to validate the authenticity and integrity of theencrypted GVM data message.

After decrypting the GVM data message (at 915), the process (at 915)sends the decrypted GVM data message along its datapath. In someembodiments, this operation entails returning a communication to the SFEport 235 (that called the decryptor to initiate the process 900) to letthe port know that the decryptor is done with its processing of the GVMdata message. The SFE port 235 can then handoff the GVM data message tothe destination GVM or can call another I/O chain operator to performanother operation on the GVM data message. After 915, the process 900ends.

When the process 900 determines (at 910) that the encryption cache 410does not store an encryption rule for the received GVM data message, theprocess 900 searches the encryption rule data store 250 to identify anencryption rule that matches the message-attribute set identified at905. If the process cannot find an encryption rule that identifies akey, and the GVM data message is encrypted (e.g., as specified by fieldin the message), the process 900 in some embodiments initiates an errorhandling process to resolve the unavailability of a decryption key fordecrypting the encrypted message. This error handling process in someembodiments queries the network agent 360 to determine whether it or thecontroller set stores an encryption rule for the message attribute setidentified at 905. When the agent has such an encryption rule, itprovides it to the process (at 920). However, in other embodiments, theerror handling process does not contact the network agent 360 to obtainthe key. Instead, it just flags this issue for an administrator toresolve.

Assuming that the process identifies (at 920) an encryption rule thatidentifies a key, the process (at 925) retrieves a key identifier fromthe identified rule, uses this identifier to retrieve a key from the keydata store 415, and decrypts the received GVM data message with theretrieved key. In some embodiments, part of the decryption operation (at925) is to authenticate an ICV generated hash of the GVM message headerand payload. Specifically, when a portion of the received GVM datamessage (e.g., its physical (e.g., L3 or L4) header values, or itslogical (e.g., VNI) header values) is hashed through an ICV operationalong with the payload by the encryptor, the decryption operationverifies this portion to validate the authenticity and integrity of theencrypted GVM data message.

After decrypting the GVM data message, the process (at 925) sends thedecrypted GVM data message along its datapath. In some embodiments, thisoperation entails returning a communication to the SFE port 235 (thatcalled the decryptor to initiate the process 900) to let the port knowthat the decryptor is done with its processing of the GVM data message.The SFE port 235 can then handoff the GVM data message to thedestination GVM or can call another I/O chain operator to performanother operation on the GVM data message.

After 925, the process transitions to 930, where in the encryption cachedata store 410, it creates a record to specify that decryption key thatshould be used to decrypt GVM data message with message-attribute setsthat are similar to the set identified at 905. In some embodiments, thisrecord is addressed in the cache 410 based on a hash value of themessage-attribute set identified at 905. After 940, the process ends.

For GVM data messages that are encrypted based on dynamically detectedevents (e.g., dynamically detected connection sessions), the GVM datamessages in some embodiments include a key identifier that identifiesthe encryption key that was used to encrypt the message or thedecryption key that should be used to decrypt the message. Someembodiments use a symmetric encryption scheme in which the same key isused to encrypt and decrypt the message, or transposed versions of thesame key are used to encrypt and decrypt the message. In theseembodiments, the GVM data message can be marked with the encryption keyidentifier or the decryption key identifier as the encryption anddecryption keys are identical or related.

FIG. 10 illustrates a process 1000 that the encryptor set 215 performsto decrypt an encrypted GVM data message that includes anencryption/decryption key identifier. In some embodiments, a decryptorin the encryptor set 215 performs this operation when its correspondingSFE port 235 calls the decryptor to check whether a received GVM datamessage is encrypted, and if so, to decrypt this message. In someembodiments, the decryptor performs this process only if the headervalue of the received GVM message specifies a key identifier thatidentifies a key for decrypting the GVM message.

As shown, the process 1000 initially extracts (at 1005) the keyidentifier from the received GVM message. Next, the process uses (at1010) the key identifier to retrieve a key from the key data store 415,and then uses (at 1015) this key to decrypt the received GVM datamessage. As mentioned above, part of the decryption (at 1015) is toauthenticate the ICV generated hash (of the received GVM data message'sheader and payload) that is encrypted with the payload of the GVM datamessage. After decrypting the GVM data message, the process sends (at1020) the decrypted GVM data message along its datapath. In someembodiments, this operation entails returning a communication to the SFEport 235 (that called the decryptor to initiate the process 1000) to letthe port know that the decryptor is done with its processing of the GVMdata message. The SFE port 235 can then handoff the GVM data message tothe destination GVM or can call another I/O chain operator to performanother operation on the GVM data message. After 1020, the process 1000ends.

By applying fine-grained encryption policies, the encryptionarchitecture of FIG. 4 can easily achieve a variety of differentencryption scenarios. For instance, it can encrypt differently the datamessages for different GVMs that execute on the host. This architecturecan also differently encrypt the data messages exchanged between theGVMs of different logical networks in a data center that providesdifferent logical networks to different users (e.g., differentdepartments, different tenants, etc.) on shared a networkinfrastructure. This hypervisor-based architecture can also encryptdifferent types of data messages from the same GVM differently. Throughguest introspection, it can gather malware data, application-centricdata and contextual data, and use this information to detect securityevents or other types of events. Once such events are detected, it canthen dynamically provide the encryptors with encryption rules necessaryfor performing on-demand, attribute-based encryption of the GVM datamessages.

FIGS. 11-17 illustrate several examples of the different encryptionscenarios that the encryption architecture of FIG. 4 can achieve. FIG.11 illustrates an example in which the encryption architecture 402differently encrypts data messages from one GVM to two different GVMs.In this example, a GVM 1105 on a host 1110 sends one packet 1150 to aGVM 1115 on a host 1120, and a second packet 1152 to a GVM 1125 on ahost 1130. In this example, an encryptor 1107 on the host 1110 encryptsthe two different packets 1150 and 1152 differently. To encrypt thesepackets differently, the encryptor (1) uses packet header valuesincluding the destination identifiers (e.g., the destination IPaddresses) to identify two different encryption rules in the data store250 that provide two different key identifiers, (2) uses theseidentifiers to retrieve two different keys from the key store 415, and(3) differently encrypts the packets by using the different encryptionkeys.

Accordingly, the two secure connections between GVM 1105 and GVMs1115/1125 can differ because they are established through two differentkeys. They can also differ by the different type of encryption operationthat is performed for each connection. For instance, the secureconnection between GVMs 1105 and 1115 might be an L2 secure connection(where L2 header and payload are encrypted), while the secure connectionbetween GVMs 1105 and 1125 might be an L4 secure connection (where L4header and payload are encrypted).

It might be beneficial to differently encrypt the GVM messages that GVM1105 exchanges with GVMs 1115 and 1125, because the GVMs 1115 and 1125might be used by different entities with which the GVM 1105 should havedifferent secure connections. Also, the GVMs 1115 and 1125 can bedifferent types of machines (e.g., one can be a webserver, while theother is a database server) that require different secure connectionswith the GVM 1105. In fact, the secure connections between the twodifferent pairs of machines might need to be two different types ofconnections (e.g., an L2 secure connections versus an L4 secureconnections).

FIG. 12 illustrates another encryption scenario in which thecommunication between one pair of GVMs is encrypted differently than thecommunication between another pair of GVMs. Specifically, in thisexample, the packets that are exchanged between GVMs 1105 and 1115 onhosts 1110 and 1120 are encrypted while the packets exchanged betweenGVMs 1205 and 1210 on hosts 1110 and 1130 are not encrypted. This occursbecause while the encryptor 1107 on the host 1110 identifies anencryption rule that provides a key for encrypting the communicationbetween GVMs 1105 and 1115, the encryptor 1207 on this host 1110 doesnot identify an encryption rule that provides a key for encrypting thecommunication between GVMs 1205 and 1210. The communication between GVMs1205 and 1210 might not be encrypted because it is not as sensitive asthe communication between GVMs 1105 and 1115, which needs to beencrypted. For instance, the users that are logged onto the GVMs 1205and 1210 might not need their communications encrypted.

FIG. 13 illustrates an example where the communication between differentpairs of GVMs is encrypted differently because the different pairs ofGVMs operate in different logical networks. This example is identical tothe example illustrated in FIG. 12 except that the communication betweenGVMs 1105 and 1115 is encrypted by a first encryption key that is usedto encrypt communication in a first logical network, while thecommunication between GVMs 1205 and 1210 is encrypted by a secondencryption key that is used to encrypt communications in a secondlogical network. In some embodiments, the encryptors 1107 and 1207identify the encryption rules that specify the encryption keys based onthe logical network identifiers (e.g., the VNIs or VDRIs) that arespecified in the packets that are sent from the GVMs 1105 and 1205.

To illustrate the concept of logical network constructs, FIG. 14illustrates an example of several logical switches that are defined bymultiple software switches that execute on multiple hosts. Specifically,this figure illustrates eight GVMs (GVM 1 to GVM 8) that execute on twohosts 1405 and 1410, which include two software switches 1415 and 1420,respectively. As shown, the two software switches 1415 and 1420implement three logical switches 1425, 1430, and 1435 that connect threesets of GVMs for three different entities (e.g., three differenttenants). Logical switch 1425 connects GVMs 1 and 4 of host 1405 and GVM6 of host 1410, logical switch 1430 connects GVM 2 of host 1405 and GVM5 of host 1410, and logical switch 1435 connects GVMs 7 and 8 of host1410 and GVM 3 of host 1405.

In hypervisors, software switches are sometimes referred to as virtualswitches because they are software and they provide the GVMs with sharedaccess to the PNIC(s) of the host. However, in this document, softwareswitches are referred to as physical switches because they are items inthe physical world. This terminology also differentiates softwareswitches from logical switches, which are abstractions of the types ofconnections that are provided by the software switches. There arevarious mechanisms for creating logical switches from software switches.VXLAN provides one manner for creating such logical switches. The VXLANstandard is described in Mahalingam, Mallik; Dutt, Dinesh G.; et al.(2013-05-08), VXLAN: A Framework for Overlaying Virtualized Layer 2Networks over Layer 3 Networks, IETF.

As mentioned above, the encryption architecture 402 of FIG. 4 can alsoimplement fine-grained, flow-based encryption rules. To illustrate thiscapability, FIG. 15 illustrates an encryption scenario in which twodifferent flows between the same pair of GVMs are encrypted differently.In this example, the encryptor 1107 encrypts a first flow between GVMs1105 and 1115 differently than a second flow between these GVMs. In somecases, the encryptor just uses different keys to encrypt these twodifferent flows. In other cases, the encryptor uses different encryptionschemes for these flows (e.g., an L2 encryption scheme for one flow andan L4 encryption scheme for another flow, or a first encryptionalgorithm for one flow and a second encryption algorithm for the otherflow).

To identify the different encryption rules for the different flows, theencryptor in some embodiments uses each flow's L2-L4 header values toidentify an encryption rule with a matching attribute set. In some ofthese embodiments, the source port identifier helps differentiate twoflows that have otherwise identical header values. In other words, thesource port identifier can serve to distinguish two different connectionsessions between two GVMs.

Also, as mentioned above, the event-detecting SVM 465 detects one ormore events relating to the flows, and in response, has the encryptionagent 360 insert or remove encryption rules from the rule data store250. As mentioned above, the event-detecting SVM 465 obtains real-timeguest-introspection (GI) data, in order to detect one or more eventsrelating to the flows, so that it can start the process for theinsertion of the encryption rules.

As further discussed above, the malware-detecting SVM 220 obtains GIdata in order to detect malware on the GVMs. Upon detecting malware,this SVM assigns the appropriate malware tag to the GVM in a data storethat it maintains. In response to this tagging, the encryption agent 360pushes encryption rules into the encryption data store 250 that causethe encryptor set 215 to encrypt the messages sent from the infected GVMand/or to encrypt the messages sent from other GVMs on the same host asthe infected GVM.

FIGS. 16 and 17 present two examples that illustrate dynamicallyencrypting packets after the detection of malware. Each of theseexamples is illustrated in three operational stages 1605-1615 or1705-1715. In the example illustrated in FIG. 16, the messages sent fromthe infected GVM are encrypted, while in the example illustrated in FIG.17, the messages sent from the other GVMs are encrypted.

The first stage 1605 in FIG. 16 shows the packets of the GVM 1650 on ahost 1655 being sent out without being encrypted. The second stage 1610then shows that the SVM 220 detects malware on GVM 1650. In response tothis detection, the SVM identifies in its data store the GVM 1650 asbeing infected. Once the encryption agent notes this designation, itpushes an encryption rule into the data store 250 to designate thatmessages sent from GVM 1650 have to be encrypted. In some embodiments,this encryption rule is indexed based on one or more attributes of theGVM 1650 (e.g., its IP address, its MAC address, etc.).

The third stage 1615 then shows that because of this encryption rule,the encryptor 1607 encrypts the data messages sent from the GVM 1650.When a GVM is infected with malware, it is useful to have its outgoingmessages encrypted in case the malware gathers data from the GVM andsends this data to undesired destinations.

In FIG. 17, the first two stages 1705 and 1710 are identical to thefirst two stages 1605 and 1610 of FIG. 16. However, in response to themalware detected in the second stage 1710 of FIG. 17, encryptionpolicies that direct the behavior of the encryption agent in the eventof an infected GVM, direct the encryption agent to push one or moreencryption rules into the data store 250 to designate that messages sentfrom all uninfected GVM on the same host to be encrypted. In someembodiments, each such encryption rule is indexed based on one or moreattributes of the uninfected GVMs.

The third stage 1715 of FIG. 17 then shows that because of theencryption rule(s), the encryptor 1707 encrypts the data messages sentfrom the GVM 1750 that is on the same host as the GVM 1650. When a GVMis infected with malware, it is useful to have messages sent from theother GVMs encrypted in case the malware tries to access the data of theuninfected GVMs through the share infrastructure (e.g., in the hostmemory or disk storage).

FIGS. 18 and 19 illustrate two processes 1800 and 1900 that are employedin some embodiments to dynamically specify encryption rules for detectedmalware. In some embodiments, the process 1800 is performed by themalware-detecting SVM 220. This SVM performs this process each time thatit receives guest introspection data from a GI thin agent 430 (e.g.,from the file and system introspectors 505 and 510) that is installed ona GVM. In some embodiments, the received GI data is in response to aquery from the SVM 220. In other embodiments, the introspecting agent430 periodically provides the GI data to the SVM 220.

As shown, this process 1800 starts (at 1805) when the SVM 220 receivesthe GI data from a thin agent 430 that is installed in a GVM. Next, theprocess determines (at 1810) whether the received GI data indicates thatthe GVM has been infected by malware. Any of the common anti-virustechniques that are used to detect the existence of malware can be usedto perform the malware detection at 1810. In fact, the malware-detectingSVM 220 in some embodiments is a commercially available third-party SVMthat scans GVMs in a hosted environment for malware. In someembodiments, examples of these third-party SVMs are SVMs that operatewith the vShield Endpoint framework that is provided for the ESXhypervisor environment by VMware Inc.

When the received GI data does not indicate the existence of malware,the process 1800 ends. On the other hand, when the process determines(at 1810) that the received GI data indicates the existence of malware,the process assigns a malware tag to the GVM in the GVM-securitydesignation data store 440, and then ends. In some embodiments, theprocess 1800 assigns one of several different security tags based on oneof several different levels of potential security conditions that itdetects based on its analysis of the received GI data. In someembodiments, some of these different security tags indicate differentlevels of confidence in the SVM's security assessment. In these or otherembodiments, some of the different security tags indicate differentdetected security conditions.

The process 1900 of FIG. 19 conceptually illustrates a sequence ofoperations that the encryption agent processor 450 performs in someembodiments. The processor 450 performs this process each time that itdetects a new security tag for a GVM that is executing on theprocessor's host. In some embodiments, the processor detects this tagwhen it is called by the malware-detecting SVM 220. In otherembodiments, the processor 450 detects this tag because it gets acallback (for which the processor 450 previously registered) from thesecurity designation data store 440. In still other embodiments, theprocessor 450 periodically scans the security designation data store 440to identify security tags that have recently been added or modified fora GVM.

As shown, this process 1900 starts (a 1905) when the encryption agentprocessor 450 detects a new security tag for a particular GVM that isexecuting on the processor's host. Next, the process determines (at1910) whether one or more encryption rules need to be specified toaddress the detected security condition on the particular GVM. To makethis determination, the encryption agent processor 450 examines itsencryption rule data store 425 to determine if it already contains theneeded encryption rule(s), and if not, then it examines the encryptionpolicies that it stores in its policy data store 456, in someembodiments. In other embodiments, the processor 450 makes thedetermination at 1910 by relaying the security tag to the virtualizationcontroller set 320, and having the virtualization controller setdetermine whether one or more encryption rules need to be specified. Instill other embodiments, the processor initially checks its own ruledata store 425 and/or policy data stores 456 for a newly detectedsecurity tag, and only contacts the virtualization controller set forthis tag when it does not have an encryption policy for this tag. In yetother embodiments, the processor only checks either the rule data store425 or the policy data store 456 for a newly detected security tag. Forinstance, in some embodiments, the processor does not have an encryptionpolicy data store 456 as encryption policies are not stored on the host.In some of these embodiments, the processor only checks (at 1910) itsencryption rule data store 425 when a new security condition isdetected.

Also, in some embodiments that employ multiple different security tagsto specify multiple different security conditions, the encryptionpolicies (of either the data store 456 or the virtualization controllerset 320) may dictate GVM data message encryption for some of thesecurity tags but not other security tags. This is because some of thesecurity tags indicate security conditions that are severe enough towarrant the encryption of the GVM data messages, while other securitytags do not.

When the process 1900 determines (at 1910) that it does not need tospecify one or more encryption rules to address the newly detectedsecurity condition of the particular GVM, the process ends. On the otherhand, when the process determines (at 1910) that it needs to specify oneor more encryption rules, the process creates (at 1915) the encryptionrule(s), stores (at 1915) the encryption rule(s) in its encryption ruledata store 425, and then ends. From the encryption rule data store 425,any newly stored encryption rule gets published by the encryption rulepublisher 454 to the encryption rule data store(s) 250 of anyencryptor/decryptor that has to enforce the rule, as mentioned above.

Some embodiments encrypt the data messages of an infected GVM upondetecting its infection and/or encrypt the data messages of other GVMson the same host as the infected GVM after detecting the infection. Toencrypt data messages from the infected GVM, the process 1900 creates(at 1915) encryption rule(s) that direct the infected GVM's encryptor215 to encrypt the GVM's data message. To encrypt data messages from theother GVMs on the same host as the infected GVM, the process 1900creates (at 1915) encryption rule(s) that direct the encryptor(s) 215 ofthe other GVM(s) to encrypt the data messages from the GVM(s). Asmentioned above, each generated encryption rule in some embodimentsidentifies its encryption process and/or its encryption key. Eachencryption rule also has a rule identifier set 620 that contains one ormore header values that uniquely identify the GVM with the data messagesthat have to be encrypted.

FIGS. 20 and 21 illustrate two processes 2000 and 2100 that are employedin some embodiments to dynamically specify encryption rules fordynamically detected flow based events. In some embodiments, the process2000 is performed by the event-detecting SVM 465. As shown, the process2000 starts (at 2005) each time that the SVM 465 receives guestintrospection data from the network introspector 515 of the GI agent 430that is installed on a GVM. As mentioned above, each time the GVM'sTCP/IP stack initiates or terminates a connection request, the stackcalls the network introspector 510. Through these calls, the networkintrospection module captures (1) every new connection request that ismade by an application that is operating on the GVM, and (2) contextualinformation (e.g., user identity, application context, etc.) for the newconnections.

The network introspector 515 provides the captured connection sessionsand their associated contextual metadata to the flow-based eventdetecting SVM 465. In some embodiments, the SVM 465 then examines itsconfiguration and cache stores to determine (at 2010) whether thecaptured event is an event for which it should provide a notification tothe encryption agent 360, and if so, whether it has previously providedsuch a notification to the encryption agent 360.

When the SVM 465 determines that the received event data does not needto be reported, the process 2000 ends. On the other hand, when the SVM465 determines that the received event data has to be reported, the SVM465 notifies (at 2015) the encryption agent 360 of the receivedflow-based event and then ends. The agent then examines the encryptionpolicies that it or the virtualization controller set 320 maintains todetermine whether it or the virtualization controller set 320 needs tocreate and push encryption rule(s) to the encryption data store 250 toaddress the detected flow-based event. When encryption rule(s) arepushed, the encryptor set 215 then encrypts the data messages related tothe detected flow-based event by using the key(s) specified in therule(s).

The process 2100 of FIG. 21 conceptually illustrates the operation ofthe encryption agent processor 450 each time that the SVM 465 reports anew flow-based event. As shown, this process 2100 starts (at 2105) whenthe encryption agent processor 450 receives a new flow-based event fromthe SVM 465. Next, the process determines (at 2110) whether one or moreencryption rules need to be specified to address the detected event forthe particular GVM on which the flow-based event was detected. To makethis determination, the encryption agent processor 450 examines itsencryption rule data store 425 to determine if it already contains theneeded encryption rule(s), and if not, then it examines the encryptionpolicies that it stores in its policy data store 456, in someembodiments. In other embodiments, the processor 450 makes thedetermination at 2110 by relaying the event data to the virtualizationcontroller set 320, and having the virtualization controller setdetermine whether one or more encryption rules need to be specified. Instill other embodiments, the processor initially checks its own ruledata store 425 and/or policy data stores 456 for a newly detected event,and only contacts the virtualization controller set for this event whenit does not have an encryption policy for this event. In yet otherembodiments, the processor only checks either the rule data store 425 orthe policy data store 456 for a newly detected event.

When the process 2100 determines (at 2110) that it does not need tospecify one or more encryption rules to address the newly detected eventon the particular GVM, the process ends. On the other hand, when theprocess determines (at 2110) that it needs to specify one or moreencryption rules, the process creates (at 2115) the encryption rule(s),stores (at 2115) the encryption rule(s) in its encryption rule datastore 425, and then ends. From the encryption rule data store 425, anynewly stored encryption rule gets published by the encryption rulepublisher 454 to the encryption rule data store(s) 250 of anyencryptor/decryptor that has to enforce the rule, as mentioned above.Each generated encryption rule in some embodiments identifies itsencryption process and/or its encryption key. Each encryption rule alsohas a rule identifier set 620 that contains one or more header valuesthat uniquely identify the flow (associated with the detected event)that has to be encrypted.

FIG. 22 presents a more-detailed diagram of an encryption system 2200 ofsome embodiments. In this system, the encryption agent 360 interactswith several servers and/or appliances that create the logical networks,manage the encryption operations within these networks, and manage thekey distribution and key refresh operations for the encryptionoperations. As shown, these servers and/or application include a networkcontroller set 2250, a network manager set 2255, and a key manager set2260.

FIG. 22 also illustrates a context-aware, hypervisor-based encryptionarchitecture 2200 that is similar to the architecture 402 of FIG. 4.Specifically, in each host, this architecture 2200 includes anencryption agent 360, an encryptor set 215, malware and event-detectingSVMs 220 and 465, and thin GI agent 430. The representation in FIG. 1 isslightly different than the representation in FIG. 4, in that in FIG. 1,the GVMs and SVMs are shown outside of the hypervisor 405, while in FIG.4, these VMs are shown to operate on top of the hypervisor 405. Bothrepresentations are accurate as these VMs operate outside of thehypervisor in a software layer that is above the hypervisor. Also, inthe architecture diagram in FIG. 1, some of the components (like the SFE210, the SFE ports 230 and 235) are not shown, in order to simplify thisfigure.

The encryption architecture 2200 leverages guest introspection toidentify security and non-security events to create context awarelogical private networks. As described above by reference to FIGS. 4 and5, guest introspection provides information about sensitive data, theapplications and the users accessing them, and the timing and manner ofthe data flows in the network. This GI-enabled encryption architectureidentifies the endpoints between which the network traffic needs to besecured, provides a way to isolate the traffic cryptographically, andprotects the traffic from other machines that share the common networkand compute infrastructure.

Guest introspection (e.g., its network introspector 515) providesconnection data to the event-detecting SVM 465, which, in turn, relaysthis information to the encryption agent 360. Guest introspection (e.g.,its file and systems introspectors 505 and 510) also provides file andsystem data to the malware-detecting SVM 220, which, in turn, definesthe security posture of the GVMs. For a defined security posture (e.g.,an infected machine), the encryption agent might supply the encryptorset 215 with one or more encryption rules that require this set toencrypt the data messages of the infected machine and/or of all the GVMsthat operate on the same host as the infected machine.

In some embodiments, the security postures are provided through ahypervisor-based framework that allows SVMs from different vendors togather GI data from the GVMs that operate on the hypervisor and providesecurity tags that define the security posture of these GVMs. Using thesecurity tags applied on a GVM, a third-party SVM can provideinformation about what it detects on the GVM, in order to allow othersecurity modules to take action with respect to the GVM. For instance,if an SVM's antivirus scan detects the presence of malware on GVM thatthe SVM is unable to clean, the SVM can tag the GVM as infected. Afirewall agent or SVM that operates on the hypervisor, can then act onthis tag and network-quarantine the infected GVM. Similarly, theencryption architecture can act on this tag to encrypt the messages ofthe infected machine and/or of all others GVMs on the same host.

The network controller set 2250 includes one or more controllers thatdefine logical overlay networks in a multi-tenant environment in whichmultiple tenants share common compute (host) and network infrastructurein a datacenter. The logical networks can be thought of as networksconnecting users and applications, isolated from the physical switchesand routers. For applications, these networks can provide point-to-pointconnections. They can be L2, L3, or L4 networks with two or moreendpoints.

By using fine-grained, encryption policies, these logical networks canbe made into logical private networks (LPNs). Each LPN can have one ormore micro-segments and each micro-segment can each be secured based ona specific security policy for that micro-segment. For instance, someembodiments encrypt connections between web servers and app servers of alogical network by using a first encryption key and a first encryptionprocess, while encrypting connections between the app servers anddatabase servers of the logical network by using a second encryption keyand a second encryption process. Some of these embodiments identify thedifferent encryption processes and different keys for the differentnetwork segments based on different header portions of the GVM datamessages exchanged between the GVMs of these network segments. Forexample, some segments can be identified based on L2 and L3 parametersonly, while other segments can be identified based on L4 parameters.

In some embodiments, the LPN are established by using EncapsulatingSecurity Payload (ESP) frame format. ESP is a security encapsulationthat provides privacy, integrity, non-repudiation and authenticity ofthe payload. ESP protocol contains a Security Parameters Index (SPI)field that can uniquely identify the security properties of the LPN(e.g., SPI identifies the encryption key for the destination of a GVMdata message). ESP is widely used in IPSec inserted between IP andTCP/UDP. In some embodiments, ESP is inserted in appropriate layers toprovide L2, L3 or L4 LPNs. ESP offload is supported by various NICs suchas Intel Kawela and Intel Niantic.

In some embodiments, the ESP encapsulation uses encryption andauthentication to provide data privacy, integrity and authenticity. Forinstance, in some embodiments, the ESP encapsulation uses AESGCM (GaloisCounter Mode) in 128 bit or 256 bits, which provides both encryption andauthentication, has better performance in contrast to the previousstandard AES-128-CBC for encryption and SHAI-HMAC for authentication. Insome embodiments, the ESP encapsulation not only encrypt the payload ofthe GVM message, but also encrypts a hash value that is generated froman integrity check value (ICV) calculations on the GVM message payloadand on some or all of the unencrypted header value (e.g., L3/L4 headervalues, or logical network identifier values) of the GVM data message.After decrypting the payload along with the ICV-generated hash, thedestination host can then authenticate the GVM data message byauthenticating the hash value for the GVM data message's header valueand payload.

In some embodiments, the logical overlay and security encapsulation(e.g., the ESP encapsulation) is performed by the encryptor set 215,which, as mentioned above, captures the traffic originating from a GVM.Through the LPN controller, the security policies and specifications(e.g., as encryption algorithm and key size) are provided by the LPNmanager 2210 of the network manager 2255, while the encryption is doneby the encryptor set 215. For VXLAN overlay network, the LPNs can secureL2 (e.g., the encryption rules are selected based on L2 parameters). Fortraffic between a web server and a database server, the LPNs can secureL4 (e.g., the encryption rules are selected at least partially based onL4 parameters). Accordingly, the encryptor set 215 in some embodimentscaptures the outbound traffic of the GVM, applies the appropriatesecurity policy, and encrypts the traffic. For incoming traffic on thedestination GVM, the encryptor set 215, in some embodiments, terminatesthe security overlay encapsulation.

In some embodiments, the encryptor set 215 is also responsible forauthenticating the GVM VNICs with the LPN controller 2220 to restrictthe member subscription into an LPN. In some embodiments, anencryptor/decryptor will be inserted for all VNICs on all the GVMs oneach hypervisor. When the encryptor/decryptor is inserted on a VNIC, theencryptor/decryptor registers certain information (e.g., the VNICinformation, VM identifier, host identifier, etc.) with the LPNcontroller 2220. The encryptor agent 360 in some embodiments is a userspace daemon that acts on behalf of the encryptor set 215 to communicatewith the LPN manager 2210, the network controller(s) 2250, and the keymanager(s) 2260.

In some embodiments, an LPN controller 2220 in the network controller2250 manages multiple LPNs in a datacenter. In some embodiments that usemore than one network controller 2250, each network controller controlsa different set of logical networks, and its LPN controller 2220configures the encryption within the logical networks of its networkcontroller 2250. The LPN controller 2220 is responsible forauthenticating and providing configuration information to the encryptorset 215. When the key needs to be rotated for an LPN, the LPN controllerensures that all the encryptor sets 215 on all of the LPN's hosts aresynchronized. The LPN controller 2220 distributes keys to the encryptorset if a key manager is not available (e.g., in a deployment that doesnot use a key manager appliance) or is not accessible by the encryptionagent (e.g., because of firewall configurations, etc). When the keymanager is not accessible, the LPN Manager in some embodiments fetchesthe keys and distributes them to the encryption agent through the LPNcontroller.

If required, the LPN controller can send kill commands to an encryptorset to destroy the key material, and sink all sensitive traffic. In someembodiments, the LPN controller sends the kill commands at the directionof the LPN manager 2210. When an LPN manager determines that a specifichost has been compromised to a particular extent that would require itsoperations to be temporarily halted, the LPN Manager can send the killcommand to that host's encryption agent 360 through the LPN controller2220. Through this controller, the LPN manager also sends a kill commandto a host, when the LPN manager determines that the compute cluster orthe datacenter in which the host operates is under severe attack so asto not be trustworthy anymore. On receiving the kill command from theLPN controller, the encryption agent sets up a policy to sink all thenetwork traffic, and propagates rules regarding the same to theencryptors 215. These rules cause the agent and/or encryptors to purgethe keys that are being used, and cause the encryptors to drop the GVMdata messages to prevent the network traffic from flowing.

The LPN manager 2210 collects the statistics from the host encryptoragents 360, which collects these statistics regularly from the encryptorsets 215 of the hosts. The LPN manager 2210 in some embodiments allows asecurity administrator to define security policies and specifications(e.g., as encryption algorithm and key size). The network managerincludes a policy engine 2215 that assists the LPN manager 2210 withthese operations. In some embodiments, the LPN manager 2210 offloads themanagement of the keys to the key manager(s) 2260 while it maintains thesecurity credentials to access the keys. The LPN manager in someembodiments creates the keys required for an LPN on the designated keymanager as per the defined security policies, but has the key managerset 2260 as the resource from which the host encryption agents pull thekeys.

In some embodiments, the key life cycle management is performed by theLPN manager 2210. The key life cycle includes various states, which insome embodiments includes: pre-active, active, deactivated, compromised,destroyed, and destroyed compromised. These states are described inManagement Interoperability Protocol Specification Version 1.2. Editedby Kiran Thota and Kelley Burgin, 9 Jan. 2014, OASIS CommitteeSpecification Draft 01/Public Review Draft 01. This specification can befound at http://docs.oasis-open.org/kmip/spec/v1.2/kmip-spec-v1.2.html.

As further described below, the LPN manager 2210 creates a key ring withmultiple active keys for an LPN (e.g., for one tenant in a multi-tenantdatacenter). In some embodiments, the LPN manager 2210 is responsiblefor the key rotation operation that adds new active keys to an LPN's keyring or remove keys that need to be inactive from the LPN's key ring. Inother embodiments, the key rotation responsibility is offloaded to theLPN controller 2220, as further described below. However, even in someof these embodiments, the LPN Manager will decide when to rotate the keybut will offload the key rotation commands to the LPN Controller whilepassing it the key identifier(s).

While the key life cycle management is performed by the LPN manager 2210in some embodiments, one of ordinary skill will realize that in otherembodiments the key manager(s) 2260 perform this operation. The keymanager set 2260 includes one or more key managers. In some embodiments,a key manager is an enterprise key manager (KM) appliance (e.g., ahardware appliance, or software module hardened for security), and it isresponsible for storing and managing a large number of keys. When theencryption system 2200 is used in a multi-tenant datacenter, severaldifferent key managers provide key-management operations for severaldifferent tenants in the datacenter. In other words, each of severaldifferent tenants has their own unique set of one or more key managersthat the tenant does not share with any other tenant. In someembodiments, however, one key manager might be shared between multipletenants.

In some embodiments, a key manager specifies constructs and mechanismsto define groups of keys for manageability, and provides varioussecurity controls (e.g., access control and authentication) to accesskeys. In some embodiments, the authentication mechanisms include: publickey infrastructure (PKI) certificates, username/password, and sharedsecrets. PKI is a type of key management system that uses hierarchicaldigital certificates to provide authentication and public keys toprovide encryption. PKI binds public keys with respective identities bymeans of certificate authority (CA). The user identity is unique in eachCA domain.

The key manager of some embodiments also enforces attestation of therequester to address the malicious requester threats. In someembodiments, the key managers use Key Management InteroperabilityProtocol (KMIP) interface as the communication protocol between keymanagement systems and encryption systems. The key managers use thisprotocol for allowing the encryption controllers/managers to managekeys, to manage key life cycles, and to register clients. Moreinformation about the KMIP can be found in publicly distributeddocument, Key Management Interoperability Protocol Specification Version1.2. Edited by Kiran Thota and Kelley Burgin, 9 Jan. 2014, OASISCommittee Specification Draft 01/Public Review Draft 01. Thisspecification can be found athttp://docs.oasis-open.org/kmip/spec/v1.2/kmip-spec-v1.2.html.

In some embodiments, the key manager set 2260 manages all the keys forone or more entities (e.g., tenants, companies, departments, etc.). Eachkey is identified by the universally unique identifier (UUID) generatedby the key manager set 2260 when the key is created. The LPN controller2220 associates an SPI with each key and informs the encryption agents360 of the mapping information. The encryptor set has a set (or ring) ofactive SPIs—the ring size is small for smaller deployments and large forlarger deployments. The ring size affects the latencies involved incoordinating a large number of endpoints within the LPN. For a ring ofsize ‘r’, keyring[0: r] has ‘r+I’ entries of which ‘r’ are active. Giventhe latest key, key_(n), key rotation removes key_(n+1) from the keyringand marks key_(n+1) as the latest. In some embodiments, key rotationprocess entails the following three steps.

1. The LPN controller 2220 directs each encryption node (a) to prefetchkey_(n+1), the latest key from the key manager, and (b) to receivetraffic with keys from keyring[o: r−1], but transmit using only keysfrom keyring[: r−1] (i.e., without the key to be removed). Allencryption nodes (where the encryptor nodes are the encryptors 215 thatare associated with an LPN) send the LPN controller an acknowledgemessage. After the key_(n+1) has been fetched, the nodes place the newkey in keyring[r] slot.

2. The LPN controller 2220 tells each node to receive using keys fromkeyring[1: r], and confirm if key_(n+1) has been fetched. At this stage,the nodes still transmit using keys from keyring[: r−1]. All nodes sendthe LPN controller an acknowledge message.

3. The LPN Controller 2220 tells each node to transmit using keys fromkeyring[1: r]. All nodes remove the key from keyring[0]. All nodes sendthe LPN controller an acknowledge message, thereby confirming keyrotation.

The above-described key rotation process provides a reliable mechanismfor rotating to a new key in a system that suffers from delay insynchronizing the key switch over of multiple nodes. This delay can besubstantial (e.g., many minutes or even into hours) when the number ofnodes is very large. Thus, this mechanism provides a scheme formaintaining an old key as a viable key in the keyring for receivingtraffic, until all the nodes can confirm receiving the new key that willtake the place of the old key in the keyring.

At any time the goal is to have a single key that is considered good,but the keyring can include multiple other keys for an LPN to use. Onebenefit of having multiple potential keys is that the encryption systemmight have to changes keys at a rate faster than the rate that new keyscan be added in a stable manner to the keyring for the LPN. Forinstance, the LPN key might have to be changed after 10 minutes (e.g.,because a threshold amount of data, such as 1 terabytes of data, havebeen encrypted with the current key), but it might take 20 minutes toadd a new key to the keyring because a very large number of nodes (e.g.,a 1000 encryptor sets). Thus, having extra keys in the keyring wouldallow the LPN controller to direct the hosts to rotate to a new key inthe keyring after the 10 minute period, even though it will take the LPNcontroller 20 minutes to add a new key to the keyring.

In some embodiments, the LPN controller directs all the LPN encryptors215 to use a new key by sending them the SPI for the new key to use,where the SPI (1) identifies the key in the keyring of each node, and(2) identifies for the destination of an encrypted GVM data message thekey that was used to encrypt the message. Alternatively, the LPNcontroller directs all the LPN encryptors to jump to the next key intheir keyrings.

In some embodiments, the encryption architecture 2200 has severalfeatures and operations to ensure its security. First, in someembodiments, the guest agent integrity is ensured by a userworld processin the hypervisor. The userworld process checks that the guest agent hasbeen loaded, code memory pages unmodified, and prevents attempts totamper with the code, by leveraging APIs to read/write guest memory,installing memory traces for read/write/execute and accessing guest CPUregisters and state.

Second, in some embodiments, the encryptor set 215 and encryption agent360 are packaged together and installed on the hypervisor together. Insome embodiments, a package integrity mechanism provides the softwareintegrity guarantee and authenticity guarantee (they are signedpackages). Third, through the encryption agent 360, the encryptor set215 registers the VNIC identifier, the GVM identifier, the hostidentifier and other information with the LPN controller 2220 in someembodiments. The LPN manager 2210, in turn, confirms with the computeinventory to make sure that the GVM and VNIC are indeed running on thespecified host. In some embodiments, the LPN manager 2210 also confirmsthe encryptor set 215 authenticity based on the host's certificate(e.g., host SSL certificate thumbprint).

Fourth, in some embodiments, the encryption agent 360 communicates withkey manager set 2260, the LPN manager 2210 and the LPN controller 2220by using client-server authentication processes (e.g., SSL/TLSprocesses). These processes authenticate the agent, manager(s) andcontroller, and encrypt messages between these authenticated modules.

Fifth, in some embodiments, the encryption agent 360 fetches the keyfrom the key manager 2260 only when required. In some embodiments, theencryptor set 215 stores the key only in memory (e.g., in volatilememory, and not on disk) and periodically refreshes the key from the keymanager as specified in the key-management policy. When the encryptorset is removed from a VNIC, the key in some embodiments is zeroed inmemory and it unregisters with the LPN manager 2210. The encryptor set215 is typically removed from a VNIC on a host when its associated GVMpowers down or when the GVM migrates to another host.

Sixth, to prevent the traffic from cryptanalytic attacks, the keys arerotated in some embodiments. In some embodiments, the key rotation isperformed periodically. In other embodiments, the key rotation is basedon the amount of data that was encrypted by using a particular key(e.g., is rotated after a terabyte of data is encrypted with the currentkey). When the encryption-key usage that is collected by the LPNcontroller 2220 shows that the usage has exceeded a threshold amount,the LPN controller 2220 in some embodiments performs the key-rotationoperation.

It should be noted that key rotation is different than refetching keys.Key rotation involves changing the encryption key for an LPN, whilerefetching is fetching the same key. As mentioned above, the LPNcontroller directs the encryptors 215 of an LPN to switch to a new keyin their keyrings by providing the SPI for the new key to use, or bydirecting them to use the next key in their keyrings. The LPN controllermight perform this action after noting that a threshold amount of data(e.g., 1 terabytes of data) has been encrypted with the current key.

On the other hand, a key is refetched in some embodiments when theencryption agent on the hypervisor is reloaded or is programmed to fetchthe same key periodically (e.g., every day), so that the audit logs onthe key manager show the continued usage of this key. This is because insome embodiments, when a VM powers down, the key is destroyed byencryption agent but the Key Manager is not informed that the key is nolonger in use.

FIGS. 23-25 illustrate the workflow message exchange within theencryption system 2200 of some embodiments. FIG. 23 illustrates themessages exchanged between a security administrator, the LPN manager2210 and the key manager set 2260 to create keys. In some embodiments,the security administrator interacts with the system (e.g., providesinput and reviews output) through the user interface of the networkmanager 2255. Through this interface, the security administrator addskeys (as further described below by reference to FIG. 23), and definessecurity policies and security groups (as further described below byreference to FIGS. 23 and 24). The keys are created on the key managersby the LPN manager 2210 as required.

As shown in FIG. 23, the workflow starts with the security administratorincorporating a key manager in the network management system. Thisincorporation involves the pairing of the LPN manager 2210 with itsassociated key manager. When a tenant has multiple key mangers, thispairing has to be done between the LPN manager 2210 and multiple keymanagers 2260. FIG. 23 shows that the pairing process starts with thesecurity administrator directing the LPN manager 2210 to generate acertificate authority (CA) for itself, and then registering this CAcertificate with the key manager 2260. As part of this registration, thesecurity administrator also obtains the key manager credentials (e.g.,from the key manager after registering the LPN manager credentials withthe key manager) and then registers these credentials with the LPNmanager 2210. In some embodiments, the key manager credentials includeone or more of the following key manager IP address, the key managerport to access, a public key infrastructure (PKI) certificate,username/password, shared secrets, etc., as mentioned above.

The LPN manager 2210 then sends a KMIP query request to its configuredkey manager to determine its capabilities and/or protocol mechanisms. Inresponse to this query, the LPN manager 2210 obtains a KIMP queryresponse that provides the requested information. As shown, once the LPNmanager 2210 has the key manager capabilities, the securityadministrator can then direct the LPN manager to create securitypolicies, and assign security policies to security groups. Through theKMIP interface, the LPN manager 2210 can then direct the key manager tocreate keys for the members of the security groups, as shown.

Security groups can be initially empty. In some embodiments, membershipto the security groups can be statically or dynamically defined. Guestintrospection allows grouping of GVMs into various security groups basedon dynamically-changing attributes, such as user identity, applicationcontext, and/or security posture of the GVM. Examples of such securitygroups include: (1) GVMs with users logged on from the Finance group,(2) GVMs running Apache server, (3) GVMs containing malware, (4) GVMsrunning on hosts with a GVM that has malware, (5) GVMs containingsensitive HIPAA patient data, etc.

FIG. 24 illustrates the workflow messaging for security groups (SGs)with dynamic membership. As shown, a security administrator in someembodiments directs the LPN manager 2210 to create security groups withdynamic membership by initially creating security groups that havemembership criteria that is based on dynamically-changing attributes. Asshown, the administrator then directs the LPN manager 2210 (1) to createsecurity policies with associated encryption policies and attributes,and (2) to assign the created security policies to the created securitygroups. Once the LPN manager 2210 receives the assignment of thesecurity policies to the security groups, the LPN manager 2210 suppliesthe security groups, the security policies and their association to theLPN controller 2220.

As further shown, the following events in some embodiments triggerupdates to the security membership depending on the attributes used todefine the groups: (1) a user login/logout event, (2) an applicationinitiating a connection, (3) an application listening on a particularport, and (4) a change in the security posture of a GVM (e.g., whetherit is tagged as having malware, known vulnerable software, sensitivedata, etc.).

FIG. 24 shows that each of these dynamic events is captured by a GImodule on a GVM, and data regarding these events is relayed by the GImodule to the LPN manager 2210 (e.g., through the encryption agent 360and intervening SVM 220 or 465, which are not shown in this figure).When the dynamically detected event requires GVM messages to beencrypted, the encryption agent 360 then pushes rules to the LPNencryptor 215, or the LPN controller 2220 pushes this rule if theencryption agent does not already have encryption rules or encryptionpolicies needed to address the needed encryption.

In other words, after detecting an event, the encryption agent 360reports this event to the LPN manager 2210, and examines its encryptionpolicy and rule data stores 456 and 425 to determine whether it has oneor more encryption rules for the detected event. If so, it pushes thisrule to the LPN encryptor 215. In response to the reported event, theLPN manager 2210 reports the event to the LPN controller 2220, whichthen determines whether it needs to push new encryption policies and/orrules to the encryption agent 360 to resolve the detected event. If so,the LPN controller 2220 pushes the encryption rules and/or policies tothe agent 360, which then pushes one or more encryption rules to the LPNcontroller to address the detected event.

Once GVM is moved to a security group, the security policies, includingthe encryption policy, associated with that group are automaticallyapplied to the traffic sent from and received for the GVM. When a GVM ispowered on, the encryptor set 215 registers the VNIC with the LPNcontroller 2220 via the encryption agent 360. The LPN controller 2220examines its data store to determine whether the GVM has been assigned asecurity policy based on its security group membership. If a securitypolicy is found, the LPN controller 2220 passes the correspondingsecurity group configuration and encryption policy information to theencryption agent, which then uses this information to push the correctencryption rules to the encryptor set's encryption rule data store 250in order to allow the encryptor set 215 to enforce the security policyfor this GVM. As mentioned above, the LPN controller 2220 providesupdated configuration and encryption policy information to theencryption agent 360 when the security group membership dynamicallychanges.

FIG. 25 illustrates the workflow messaging that some embodiments use toencrypt or decrypt the GVM data messages. As shown, this workflow startinitially when the encryptor 215 and the encryption agent 360 boot-up(e.g., when the host boots up) and engage in a handshake operation tovalidate their connection. Next, the encryption agent establishes aconnection with the LPN controller 2220 to get security groupconfiguration and policy tables.

When a GVM is booted-up or is migrated to this host, the encryptor isadded to the chain of I/O operations for appropriate VNICs. Theencryptor gathers the details from the VNIC and logical switch and sendthe LinkUP message to the encryption agent. The encryptor set 215 thenasks the encryption agent 360 for key information when it receives a GVMmessage that is sent from the booted-up GVM to another GVM. In response,the encryption agent 360 asks the LPN controller to authenticate theendpoint (i.e., to authenticate the VNIC of the GVM), and provide keymanager credentials and the lookup tables. Upon receiving key managercredentials and the lookup tables, the encryption agent 360 contacts thekey manager specified by the LPN controller and obtains the needed key.The encryption agent then passes the requested key(s) to the encryptorset 215 (e.g., in a lookup table that includes the key(s)).

As shown, the encryption agent periodically or on-demand sends theencryptor set 215 with Get Info requests, in order to retrievestatistics from the encryptor set about its operation. The encryptionagent then passes these statistics to the LPN controller, and based onthese statistics, the LPN controller sends updates to the encryptionagent, which relays these updates to the encryptor set 215. In someembodiments, the encryption agent periodically gathers statistics fromthe encryptor set and passes the statistics to the LPN controller, inorder to allow this controller to start the dynamic key rotationoperations, which were described above. In other words, thestat-gathering and update mechanism of FIG. 25 is used in someembodiments to rotate keys.

As mentioned above, the LPN manager 2210 also periodically or on-demandcollects statistics that the encryption agent gathers from the encryptorset 215. The LPN manager 2210 uses these statistics to visuallyrepresent the health of the LPN to the administrator/user and thestatistics can be displayed or provided as responses to REST APIs forvarious third party solutions which determine the network capacity,optimizations, etc. (e.g., to optimize performance by moving aroundGVMs). The LPN manager 2210 is the single point of interaction for theadministrators to define LPNs, to check the health status(green/yellow/red), to provide any remediation, and to performtroubleshooting/debugging etc.

Two exemplary deployments of the context-aware network encryption ofsome embodiments will now be described. The first example involves asoftware-defined datacenter (SDDC) for a hospital. This datacenter hasnumerous users who may be part of different Active Directory groups,such as Doctors and Nurses. The datacenter also runs the hospital Epicservers that store confidential patient data. In this system, theadministrator deploys security solutions to detect malware, vulnerablesoftware, and sensitive HIPAA patient data on GVMs.

The hospital has the following example security policies:

1. Security Policy 1: Only Doctors and Nurses can access patient data onEpic servers. All such accesses must be via secure and encryptedchannels.

2. Security Policy 2: Doctors' and Nurses' GVMs need to be scanned withan antivirus solution daily.

3. Security Policy 3: GVMs diagnosed with a virus or known malware risklevel higher than “medium” must have all traffic dropped.

4. Security Policy 4: GVMs with confidential patient data on their disksthat violates HIPAA policy must have all traffic encrypted.

5. Security Policy 5: All GVMs will be scanned for vulnerabilitiesweekly.

6. Security Policy 6: GVMs with vulnerabilities of CVSS score higherthan 8 must have all incoming and outgoing traffic dropped.

In order to comply with the aforementioned security policies, thesecurity administrator goes through the following process:

1. Create Security Group “Doctors” with dynamic membership criteria sothat any GVM with a doctor logged on it becomes a member, and drops outof membership when the doctor logs out.

2. Create Security Group “Nurses” with dynamic membership criteria sothat any GVM with a nurse logged on it becomes a member, and drops outof membership when the nurse logs off.

3. Create Security Group “Epic Servers” with dynamic membership criteriaso that any GVM running Epic application, and listening on a particularport becomes a member.

4. Create Security Group “UnsecuredVMs” with dynamic membership criteriabased on the security tag “HIPAA violation” on a GVM.

5. Create Security Group “Quarantine” with dynamic membership criteriabased on the security tags “Vulnerable.CVSS>8”, “MalwareFound. Medium”,or “MalwareFound.High” on a GVM.

6. Create security policies for the following criteria, and choose anencryption scheme.

(a) Doctors to Epic Servers: Encrypt

(b) Nurses to Epic Servers: Encrypt

(c) UnsecuredVMs to Any: Encrypt

(d) Quarantine to Any: Drop

7. Assign the security policies to the security groups.

When an Epic application is initiated on a GVM, the GVM becomes part ofthe “Epic Servers” security group. When a doctor (or nurse) logs on to aGVM, it becomes a member of the “Doctors” (“Nurses”) security group.When the doctor (or nurse) tries to access the Epic server, theappropriate security policy is enforced, and the traffic is encrypted asdefined by the policy. Similarly, if a regular scan of the GVM detectsHIPAA data on it, the GVM is automatically moved to the “UnsecuredVMs”SG, and all traffic to and from that GVM is encrypted. In this hospitalsystem, attributes can be used to trigger context-aware networkencryption in various scenarios. For instance, it can be extended to thegranularity of a single connection, i.e., encryption decision can bemade per connection based on the attributes of each individualconnection (such as initiating application).

The second example involves an SDDC for an online shopping place,WonderDiapers.com. This datacenter uses a typical three-tier webapplication deployment that runs webservers (Web), application (App)servers and database (DB) servers. The database servers containsensitive credit card information. The administrator deploys securitysolutions to detect malware, vulnerable software and the sensitive PCIdata on these database GVMs.

This SDDC has the following mandated security policies in someembodiments:

1. Security Policy 1: Only the Database Administrator can access DBservers. All such accesses must be via secure and encrypted channels.

2. Security Policy 2: All GVMs need to be scanned with antivirussolution daily.

3. Security Policy 3: All GVMs will be scanned for vulnerabilitiesweekly.

4. Security Policy 4: GVMs with confidential PCI data on their disksthat violates PCI policy must have all traffic encrypted.

5. Security Policy 5: GVMs diagnosed with a virus or known malware risklevel higher than “medium” must have all outgoing traffic dropped.

6. Security Policy 6: GVMs with vulnerabilities of CVSS score higherthan 8 must have all incoming and outgoing traffic dropped.

In order to comply with the aforementioned security policies, thesecurity administrator goes through the following process:

1. Create Security Group “Web” with dynamic membership criteria so thatany GVM that runs the Apache webserver application becomes a member.

2. Create Security Group “App” with dynamic membership criteria so thatany GVM that runs data-processing applications becomes a member.

3. Create Security Group “DB” with dynamic membership criteria so thatany VM that runs Oracle database becomes a member.

4. Create Security Group “DBAdmin” with dynamic membership criteria sothat any GVM with a user that belongs to the Database AdministratorGroup logged in becomes a member, and drops out of membership when theadministrator logs off.

5. Create Security Group “Quarantine” with dynamic membership criteriabased on the security tags “Vulnerable.CVSS>8”, “MalwareFound.Medium”,or “MalwareFound.High” on a GVM.

6. Create security policies for the following criteria, and choose anencryption scheme.

(a) Web Server to App Server: Encrypt

(b) DBAdmin to DB Server: Encrypt

(c) Not DBAdmin to DB Server: Drop

(d) App Server to DB Server: Encrypt

(e) Quarantine to Any: Drop

7. Assign the security policies to the security groups.

At initial deployment, the administrator deploys the Apache Webserver onseveral GVMs. Since the GVMs arc running the Apache application, theybecome part of the Web Security Group. Similarly, the App and DBSecurity Groups are populated when the three-tier deployment iscompleted. At this point, all App to DB traffic is automaticallyencrypted due to the policies above. Similarly, all Web to App SecurityGroup communication is encrypted. Also, apart from the Web SecurityGroup members, the DB Security Group can only be accessed by a DBadministrator. Any other user trying to access members of the DBSecurity Group will see that these requests are dropped.

At this point, normal operations are in progress. Assume now that thedeployed version of Apache on one of the webservers has knownvulnerabilities with CVSS scores greater than 8. When the vulnerabilityscan runs weekly, the vulnerability is detected, the GVM is tagged withthe “Vulnerable.CVSS>8” tag. The particular Webserver is then moved tothe Quarantine Security Group and it can no longer send or receivetraffic. Similarly, assume that high-risk malware is detected on one ofthe Webservers. Again, this particular Webserver GVM is tagged with the“MalwareFound.High” tag and is automatically moved to the QuarantineSecurity Group and is cordoned off for all network access.

To illustrate the dynamic addition of GVMs to security groups and theassociated dynamic application of encryption rules to the securitygroups to the GVMs, two example processes will now be described byreference to FIGS. 26 and 27. FIG. 26 illustrates a process 2600 thatvarious components of the encryption system 2200, of some embodiments,performs to dynamically add an infected GVM to a malware security groupand to dynamically encrypt the data messages from this GVM based on thesecurity group's encryption rule. After the malware infection has beenresolved, this process dynamically removes the GVM from the malwaresecurity group, and stops encrypting its data messages as the group'sencryption rule is no longer applicable to the GVM.

As shown, the process 2600 starts when the malware-detecting SVM 220detects (at 2605) malware on a GVM executing on the SVM's host. Next,this SVM assigns (at 2610) an “infected” tag to the GVM in the securitydesignation data store 440. Based on this designation, the encryptionagent processor 450 has (at 2615) the GVM added to a malware-infectedsecurity group that this processor 450 or the LPN controller 2220maintains. When the processor 450 maintains this security-group'smembership, the processor in some embodiments adds the GVM's identifierto this membership and notifies the LPN controller 2220 to do the same.Otherwise, the processor in some embodiments directs the LPN controller2220 to add the GVM's identifier to this group's membership.

Next, at 2620, the process 2600 creates an encryption rule that requiresthe encryption of the infected GVM's data messages, and then pushes thisencryption rule to an encryptor along the infected GVM's datapath (e.g.,pushes the encryption rule data store 250 of the encryptor 215 for theinfected GVM). In some embodiments, the agent processor 450 pushes thisrule by first resolving an encryption policy that it maintains, orsending the detected malware event to the LPN controller 2220, whicheither sends this agent an encryption policy to resolve or pushes anencryption rule to this agent after the controller resolved the policy.The policy that is being resolved either by the agent or the LPNcontroller is a policy that requires the encryption of the data messagessent by a malware infected GVM. After resolving this policy andgenerating an encryption rule for the malware infected GVM, theprocessor 450 stores the encryption rule in its rule data store 425,from where it is published to the encryption rule data store 250 of theinfected GVM's encryptor 215.

Once this rule is stored in the data store 250, the infected-GVM'sencryptor 215 encrypts (at 2625) the data messages from the infected GVMby using the encryption key that is identified in the encryption rule.As mentioned above, the encryption processor 450 (1) retrieves from theappropriate key manager 2260 the encryption keys for rules that itreceives or creates, and (2) stores these keys in the key data store420, from where the keys get published to the encryptor's key data store415 by the publisher 454.

At some point, a system administrator removes (at 2630) the malware fromthe infected GVM. In response, the infected tag for this GVM is removed(at 2635) from the security designation data store 440 manually by theadministrator or in an automated manner based on the malware-detectingSVM's assessment that the malware infection has been removed. Once theinfected tag is removed, the GVM is removed (at 2635) from themalware-infected security group (by the processor 450 or the LPNcontroller 2220). Next, at 2640, the encryption rule that was createdand pushed at 2620 is removed from the data stores 425 and 250. In someembodiments, the encryption agent processor 450 directs the removal ofthis rule from these data stores.

FIG. 27 illustrates a process 2700 that various components of theencryption system 2200 of some embodiments perform to add a GVM to asecurity group after a flow-based event is detected, and to remove theGVM from the security group when the flow-based event is terminated. Asshown, the process 2700 starts (at 2705) when the event-detecting SVM465 receives guest introspection data from the network introspector 515of the GI agent 430 that is installed on a GVM. The network introspector515 of some embodiments provides the captured connection sessions andtheir associated contextual metadata to the flow-based event detectingSVM 465.

The SVM 465 then examines (at 2705) its configuration and determinesthat the captured event is an event for which it should provide anotification to the encryption agent processor 450. For instance, thedetected event is a login event, and the captured contextual metadatamight show that a doctor logged into a particular application. When theSVM 465 determines that the received event data has to be reported, theSVM 465 notifies the encryption agent processor 450 of the receivedflow-based event.

The processor 450 then examines (at 2705) the security groups that it orthe LPN controller 2220 maintains, and determines (e.g., directly orindirectly through the LPN controller) that the detected flow needs tobe added to a particular security group. For example, the particularsecurity group might be for doctors who logon on to the particularapplication. Once the detected flow is added to a particular securitygroup, the process 2700 applies (at 2710) the security policy of thesecurity group to the detected flow. This application entails theencryption processor 450 or the LPN controller 2220 creating encryptionrule(s) to address the detected flow-based event and pushing theencryption rule(s) to the processor's rule data store 425. From thisdata store, the encryption rule is published (at 2715) to the encryptionrule data store 250 of the encryptor 215 of the GVM at which theflow-based event was detected. One example of such an encryption rulewould be an encryption rule that specifies a particular encryptionprocess and/or encryption key for encrypting messages that are sent forflows relating to a doctor's interaction with the particularapplication.

Once this rule is stored in the data store 250, the GVM's encryptor 215encrypts (at 2715) the data messages from the GVM by using theencryption key that is identified in the encryption rule. As mentionedabove, the encryption processor 450 (1) retrieves from the appropriatekey manager 2260 the encryption keys for rules that it receives orcreates, and (2) stores these keys in the key data store 420, from wherethe keys get published to the encryptor's key data store 415 by thepublisher 454.

From the network introspector 515, the event-detecting SVM 465 at somepoint receives GI data that indicates the end of the detected flow basedevent. After detecting the end of the flow-based event (at 2720), theprocess 2700 removes (at 2725) the encryption rule from the data store250 of the encryptor 215 of the GVM from which the flow emanated. Insome embodiments, the encryption agent processor 450 directs the removalof this rule from these data stores. Along with the encryption rule, theprocess 2700 in some embodiments removes from the processor's policydata store the encryption policy as well, if one such policy was pushedby the LPN controller.

In the above-described processes (e.g., processes 2600 and 2700),encryption rules and/or policies are sent from the LPN controller 2220to a host's encryption agent 360. One of ordinary skill will understandthat in some embodiments, security group membership and/or membershipupdates are sent from the LPN controller 2220 to a host encryption agent360. For instance, in some embodiments, the LPN controller 2220 sends toa host's encryption agent 360 a security group and an encryption policythat is applicable to the security group. The encryption policy requiresa particular encryption for a first type of data flow (e.g., the policyrequires encryption of data messages exchanged between two machines onwhich two medical practitioners (such as doctors and nurses) arelogged).

When the LPN controller 2220 later determines that an initiated dataflow on a particular GVM of the host is a first-type data flow (e.g., isa flow between a doctor that is using the GVM with a nurse on anotherhost's GVM), the LPN controller 2220 in some embodiments adds theparticular GVM to the security group, and notifies the host's encryptionagent of the addition of this GVM as a member of the security group.Based on this updated membership, the encryption agent processor 450generates an encryption rule for the encryptor of the particular GVM touse, when assessing whether the encryptor should encrypt first flow-typedata messages from the particular GVM. When the LPN controller 2220 orencryption agent processor 450 later determines the end of the detecteddata flow, the particular GVM is removed from the security group, andbased on this updated membership, the encryption rule that waspreviously provided to the encryptor is discarded, in some embodiments.In some embodiments, rules and/or policies are removed from theirrespective data stores after a period in which they do not get used.

Many of the above-described features and applications are implemented assoftware processes that are specified as a set of instructions recordedon a computer readable storage medium (also referred to as computerreadable medium). When these instructions are executed by one or moreprocessing unit(s) (e.g., one or more processors, cores of processors,or other processing units), they cause the processing unit(s) to performthe actions indicated in the instructions. Examples of computer readablemedia include, but are not limited to, CD-ROMs, flash drives, RAM chips,hard drives, EPROMs, etc. The computer readable media does not includecarrier waves and electronic signals passing wirelessly or over wiredconnections.

In this specification, the term “software” is meant to include firmwareresiding in read-only memory or applications stored in magnetic storage,which can be read into memory for processing by a processor. Also, insome embodiments, multiple software inventions can be implemented assub-parts of a larger program while remaining distinct softwareinventions. In some embodiments, multiple software inventions can alsobe implemented as separate programs. Finally, any combination ofseparate programs that together implement a software invention describedhere is within the scope of the invention. In some embodiments, thesoftware programs, when installed to operate on one or more electronicsystems, define one or more specific machine implementations thatexecute and perform the operations of the software programs.

FIG. 28 conceptually illustrates a computer system 2800 with which someembodiments of the invention are implemented. The computer system 2800can be used to implement any of the above-described hosts, controllers,and managers. As such, it can be used to execute any of the abovedescribed processes. This computer system includes various types ofnon-transitory machine readable media and interfaces for various othertypes of machine readable media. Computer system 2800 includes a bus2805, processing unit(s) 2810, a system memory 2825, a read-only memory2830, a permanent storage device 2835, input devices 2840, and outputdevices 2845.

The bus 2805 collectively represents all system, peripheral, and chipsetbuses that communicatively connect the numerous internal devices of thecomputer system 2800. For instance, the bus 2805 communicativelyconnects the processing unit(s) 2810 with the read-only memory 2830, thesystem memory 2825, and the permanent storage device 2835.

From these various memory units, the processing unit(s) 2810 retrieveinstructions to execute and data to process in order to execute theprocesses of the invention. The processing unit(s) may be a singleprocessor or a multi-core processor in different embodiments. Theread-only-memory (ROM) 2830 stores static data and instructions that areneeded by the processing unit(s) 2810 and other modules of the computersystem. The permanent storage device 2835, on the other hand, is aread-and-write memory device. This device is a non-volatile memory unitthat stores instructions and data even when the computer system 2800 isoff. Some embodiments of the invention use a mass-storage device (suchas a magnetic or optical disk and its corresponding disk drive) as thepermanent storage device 2835.

Other embodiments use a removable storage device (such as a floppy disk,flash drive, etc.) as the permanent storage device. Like the permanentstorage device 2835, the system memory 2825 is a read-and-write memorydevice. However, unlike storage device 2835, the system memory is avolatile read-and-write memory, such a random access memory. The systemmemory stores some of the instructions and data that the processor needsat runtime. In some embodiments, the invention's processes are stored inthe system memory 2825, the permanent storage device 2835, and/or theread-only memory 2830. From these various memory units, the processingunit(s) 2810 retrieve instructions to execute and data to process inorder to execute the processes of some embodiments.

The bus 2805 also connects to the input and output devices 2840 and2845. The input devices enable the user to communicate information andselect commands to the computer system. The input devices 2840 includealphanumeric keyboards and pointing devices (also called “cursor controldevices”). The output devices 2845 display images generated by thecomputer system. The output devices include printers and displaydevices, such as cathode ray tubes (CRT) or liquid crystal displays(LCD). Some embodiments include devices such as a touchscreen thatfunction as both input and output devices.

Finally, as shown in FIG. 28, bus 2805 also couples computer system 2800to a network 2865 through a network adapter (not shown). In this manner,the computer can be a part of a network of computers (such as a localarea network (“LAN”), a wide area network (“WAN”), or an Intranet, or anetwork of networks, such as the Internet. Any or all components ofcomputer system 2800 may be used in conjunction with the invention.

Some embodiments include electronic components, such as microprocessors,storage and memory that store computer program instructions in amachine-readable or computer-readable medium (alternatively referred toas computer-readable storage media, machine-readable media, ormachine-readable storage media). Some examples of such computer-readablemedia include RAM, ROM, read-only compact discs (CD-ROM), recordablecompact discs (CD-R), rewritable compact discs (CD-RW), read-onlydigital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a varietyof recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.),flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.),magnetic and/or solid state hard drives, read-only and recordableBlu-Ray® discs, ultra density optical discs, any other optical ormagnetic media, and floppy disks. The computer-readable media may storea computer program that is executable by at least one processing unitand includes sets of instructions for performing various operations.Examples of computer programs or computer code include machine code,such as is produced by a compiler, and files including higher-level codethat are executed by a computer, an electronic component, or amicroprocessor using an interpreter.

While the above discussion primarily refers to microprocessor ormulti-core processors that execute software, some embodiments areperformed by one or more integrated circuits, such as applicationspecific integrated circuits (ASICs) or field programmable gate arrays(FPGAs). In some embodiments, such integrated circuits executeinstructions that are stored on the circuit itself.

As used in this specification, the terms “computer”, “server”,“processor”, and “memory” all refer to electronic or other technologicaldevices. These terms exclude people or groups of people. For thepurposes of the specification, the terms display or displaying meansdisplaying on an electronic device. As used in this specification, theterms “computer readable medium,” “computer readable media,” and“machine readable medium” are entirely restricted to tangible, physicalobjects that store information in a form that is readable by a computer.These terms exclude any wireless signals, wired download signals, andany other ephemeral or transitory signals.

While the invention has been described with reference to numerousspecific details, one of ordinary skill in the art will recognize thatthe invention can be embodied in other specific forms without departingfrom the spirit of the invention. For instance, while theencryption/decryption processes were described above by reference to thehost architecture 200, 400, and 2200, one of ordinary skill will realizethat these processes could be implemented in a variety of differentarchitectures that capture and encrypt/decrypt GVM messages at varietyof different locations along their egress path out of the host. Forinstance, in some embodiments, the encryption/decryption processes areimplemented in the PNIC of the host. In other words, the PNIC of thehost in some embodiments examines the GVM messages to determine whetherit should encrypt/decrypting them before sending them out of the host orsending them to their destination GVMs. In some embodiments, the PNICperforms other functions such as defining logical networks (e.g.,through the user of technologies such as SRIOV).

Also, a number of the figures (e.g., FIGS. 1, 8, 9, 10, 18, 19, 20, 21,26 and 27) conceptually illustrate processes. The specific operations ofthese processes may not be performed in the exact order shown anddescribed. The specific operations may not be performed in onecontinuous series of operations, and different specific operations maybe performed in different embodiments. Furthermore, the process could beimplemented using several sub-processes, or as part of a larger macroprocess.

Much of the discussion above relates to capturing and encrypting datamessages sent from GVMs in a multi-VM hosted environment. However, oneof ordinary skill will realize that the encryption architecture of someembodiments can be used to encrypt and decrypt data messages from anydata end node (such as storage nodes, etc.). Accordingly, all of theabove-described encryption architectures (such as the architectures ofFIGS. 2, 4, and 22) and the above-described processes (e.g., of FIGS. 1,8, 9, 10, 18, 19, 20, 21, 26 and 27) are equally applicable forencrypting and decrypting data messages from arbitrary data nodes in avirtualized environments. Also, the architecture of some embodiments canbe used to provide other cryptographic services, such as signingmessages for verification purposes, etc. In view of the foregoing, oneof ordinary skill in the art would understand that the invention is notto be limited by the foregoing illustrative details, but rather is to bedefined by the appended claims.

1. For a computer that executes at least one virtual machines (VM), anencryption method comprising: from the VM, receiving different datamessages that are part of first and second data flows from the VM; andencrypting the data messages in the first flow differently than the datamessages in the second flow.
 2. The method of claim 1, whereindifferently encrypting the data messages in the first and second flowscomprises encrypting the data messages in the first flow, while notencrypting the data messages in the second flow.
 3. The method of claim1, wherein differently encrypting the data messages in the first andsecond flows comprises encrypting the data messages in the first flow byusing a first encryption process, while encrypting the data messages inthe second flow by using a second encryption process.
 4. The method ofclaim 1, wherein differently encrypting the data messages in the firstand second flows comprises encrypting the data messages in the firstflow by using a first encryption key, while encrypting the data messagesin the second flow by using a second encryption key.
 5. The method ofclaim 1, wherein differently encrypting the data messages in the firstand second flows comprises encrypting a first portion of each datamessage in the first flow, while encrypting a second portion of eachdata message in the second flow, wherein the first and second portionsare different portions of their respective data messages.
 6. The methodof claim 1, where the first and second flows have differentdestinations.
 7. The method of claim 1, where the first and second flowshave the same destination.
 8. The method of claim 7, where the samedestination is another VM on another computer.
 9. The method of claim 1further comprising: detecting start of a particular flow; definingencryption rules for specifying how the data messages of the particularflow have to be encrypted; providing the encryption rules to a set ofencryptors that (i) intercepts data messages of the particular flow andother flows, (ii) determines whether to encrypt the data messages basedon the encryption rules, (iii) encrypts the data messages upondetermining to encrypt the data messages, and (iv) foregoes encryptingthe data messages upon determining that no encryption is needed.
 10. Themethod of claim 1 further comprising inserting in each encrypted datamessage an encryption identifier, said encryption identifier forallowing a destination of the data message to decrypt the encrypted datamessage.
 11. The method of claim 10, wherein the encryption identifieris a key identifier that identifies an encryption key that was used toencrypt the data message.
 12. The method of claim 10, wherein theencryption identifier is a key identifier that identifies decryption keythat is needed to decrypt the data message.
 13. The method of claim 10,wherein the encryption identifier is an encryption identifier thatidentifies an encryption process that was used to encrypt the datamessage.
 14. The method of claim 10, wherein the encryption identifieris a decryption identifier that identifies decryption process that isneeded to decrypt the data message.
 15. The method of claim 10, whereinthe encryption identifier is inserted in a header field of the datamessage.
 16. A non-transitory machine readable medium storing a programfor encrypting messages from a virtual machine (VM) that executes on acomputer, the program comprising sets of instructions for: from the VM,receiving different data messages that are part of first and second dataflows from the VM; and encrypting the data messages in the first flowdifferently than the data messages in the second flow.
 17. Thenon-transitory machine readable medium of claim 16, wherein the set ofinstructions for differently encrypting the data messages in the firstand second flows comprises a set of instructions for encrypting the datamessages in the first flow, while not encrypting the data messages inthe second flow.
 18. The non-transitory machine readable medium of claim16, wherein the set of instructions for differently encrypting the datamessages in the first and second flows comprises a set of instructionsfor encrypting the data messages in the first flow by using a firstencryption process, while encrypting the data messages in the secondflow by using a second encryption process.
 19. The non-transitorymachine readable medium of claim 16, wherein the set of instructions fordifferently encrypting the data messages in the first and second flowscomprises a set of instructions for encrypting the data messages in thefirst flow by using a first encryption key, while encrypting the datamessages in the second flow by using a second encryption key.
 20. Thenon-transitory machine readable medium of claim 16, wherein the set ofinstructions for differently encrypting the data messages in the firstand second flows comprises a set of instructions for encrypting a firstportion of each data message in the first flow, while encrypting asecond portion of each data message in the second flow, wherein thefirst and second portions are different portions of their respectivedata messages. 21-30. (canceled)
 31. The non-transitory machine readablemedium of claim 16, wherein the set of instructions for differentlyencrypting data messages comprises sets of instructions for: for eachflow, using a set of message attribute values to identify one encryptionrule from a plurality of stored encryption rules, said identifiedencryption rule being an encryption rule that is defined for the flow;retrieving an encryption parameter for an encryption process from theidentified encryption rule; and using the identified encryptionparameter to encrypt the data messages of the flow.